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Preface 



For the twenty-second year, the Research and Theory Division of the Association for 
Educational Communications and Technology (AECT) is sponsoring the publication of 
these Proceedings . Papers published in this volume were presented at the National AECT 
Convention in Long Beach, CA. A limited quantity of these Proceedings were printed 
and sold. It is also available on microfiche through the Educational Resources 
Clearinghouse (ERIC) system. 

REFEREEING PROCESS: Papers selected for presentation at the AECT Convention and 
included in these Proceedings were subjected to a rigorous blind reviewing process. All 
references to authorship were removed from proposals before they were submitted to 
referees for review. Approximately fifty percent of the manuscripts submitted for 
consideration were selected for presentation at the convention and for publication in these 
Proceedings . The papers contained in this document represent some of the most current 
thinking in educational communications and technology. 

This volume is indexed by both author and descriptor. The index for volumes 1-6 (1979 
- 1984) is included in the 1986 Proceedings , and the index for volumes 7 - 10 is in the 
1988 Proceedings . After 1988, each volume contains indexes for that year only. 
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What Is AECT? 

The Association for Educational Communications & Technology is the leading international organization 
of its kind. The mission of the association is to provide leadership in educational communications and 
technology by linking professionals holding a common interest in the use of educational technology and its 
application to the learning process. 

Who Belongs to AECT? 

AECT members are people who need to keep up with changing technology for education and training, 
including: 

Media Specialists 
College and University Faculty 
K-12 Classroom Teachers 
Librarians 

Instructional Designers 
Corporate/Military Trainers 

How Long has AECT Been at the Forefront of Technology? 

Unique among technology organizations, AECT has provided more than 70 years of service and 
innovation. We’ve grown up with technology, advocating its integration into education from blackboards to 
interactive video and hypermedia. 

AECT began as a department of the National Education Association in 1923. In 1947, the organization 
supported educators as they adapted to train World War II service personnel. In 1974, AECT achieved 
independence. 

Today, more than 3,000 professionals rely on this organization as they embrace new methods, new 
equipment and new techniques in the fast-changing field of education and technology. 

Why Join AECT Now? 

Join AECT and youMl increase your effectiveness, your expertise and your skills. You will also enhance 
your professional standing and earning potential. All within a dynamic and growing organization, open to 
your ideas and suggestions. 

What Does AECT Offer Its Members? 

The association's broad range of advantages includes: 

• Annual International Conference with over 300 educational sessions and workshops 

• Access to Technology + Learning Exposition (in partnership with NSBA) 

• Seven special-interest divisions and two special-interest councils 

• Summer Leadership Institute 

• Acclaimed publications, such as TechTrends. Educational Technology Research & 
Development and The Quarterly Review of Distance Education 

• Discounts on books and media 

• AECT listserv discussion group 

• Membership newsletter E-News 

• Web site with the latest Association activities 

• On-line job placement listings 

• Low-cost insurance through Forrest T. Jones 

• On-line Membership Directory 

• Affinity card (with credit approval) 

• Membership ID card 

• Internships 

• Awards and scholarship programs 



Learning Resource Specialists 
Curriculum Developers 
Communications Specialists 
Education Administrators 
School Network Administrators 
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Design & Development 
Management 
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Research & Theory 
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Computer-based Cooperative, Collaborative and 
Individual Learning: A report of research 



Scott Adams 

Arkansas Tech University 



Computers and Learning Theories 

Computers have been used for years in educational settings. Schools purchased personal 
computers long before adequate educational software was available. Initially, software for these machines 
was proprietary and would only work for the brand of machine for which it was designed. It became 
obvious that in the educational marketplace, proprietary software was not the answer and the demand for 
applications of a specific educational nature increased. These were at first stand-alone applications, then 
local network applications, and finally global applications with the development of the Internet and World 
Wide Web (WWW). State agencies were given the mandate to coordinate, deliver, support and maintain 
computer and network services and occasionally to initiate them. 

Educationally, the state of Kentucky has embraced the Internet and its attendant technologies 
following the adoption of the Kentucky Educational Reform Act in 1990. Fiber optic connections link 
every school district in the state, and the Kentucky Educational Technology System (KETS) supports the 
daily use of electronic mail and instructional web pages by the school systems across the state. The 
Commonwealth of Kentucky, in the establishment of the Kentucky Educational Technology System, 
reports a student to computer ratio of almost nine students to one computer (Herrick, 1 999). 

In the United States, the Goals 2000 program gave impetus to the growth of educational 
technology. President Clinton, in his 1997 State of the Union Address, outlined this program as one of his 
priorities for the U.S. Department of Education, which included the goal of connecting every classroom to 
the Internet by the year 2000, and having a technologically literate student population (Education, 1 997). 
The President’s Committee on Science and Technology (Technology, 1997) cited a student to computer 
ratio of 125 to 1 in 1983-4 falling to a level of 10.5 to 1 for the 1994-5 school year. Over the past decade 
the introduction of computers into the classroom has accelerated — not just in Kentucky but in the nation as 
well. 

With the development of new equipment and new technologies in education, one aspect typically 
lags behind - the development of educational methods for using the technologies and the training of 
teachers in the implementation of these technologies in a classroom setting. Since in many instances 
students must share computers, methods of instruction need to be devised which allow for best learning 
practices. While many factors need to be researched still, one is the effect of prior training in skills needed 
in working together. 

One promising method of instruction is cooperative learning, which is a way of teaching with two 
or more students working together in a structured manner to pursue the same educational goal. 
Collaborative learning is a second instructional format, which has students working in groups of two or 
more to pursue the same educational goal. Individualized learning is also prominent in computer-based 
instruction. Individualized learning involves a single student pursuing an educational goal. 

In a situation involving computer-based instruction instructors may structure learning in several 
ways. One established method is to use individual computer-based instruction, which places one student, 
working alone, in front of a computer running a computer-based instructional program. A second method is 
cooperative network based instruction used in varying forms since the 1970s, 

The educational uses of the World Wide Web are an outgrowth of the uses documented in the area 
of cooperative network based instruction. Individualized network based instruction dates back another 
twenty years before the work of Okey and Majer (Okey & Majer, 1976), Saettler (1990) placed the origins 
of the computer assisted instruction movement in the 1950s. Approximately twenty years followed the 
creation of the first modem computers before the first experiments were conducted with time-sharing 
network based instructional methods. Another twenty years passed before the technology advanced to 
allow microcomputers to be connected to the Internet and the World Wide Web technology to be invented. 




The WWW has seen phenomenal growth in the years since Tim Berners-Lee originated the basic concepts 
for the World Wide Web (Berners-Lee, 1999), 

The purpose of this study was to determine whether collaborative, cooperative or individual 
treatment elicits the best performance. The primary goal of this study was to determine the effects of 
cooperative and collaborative learning environments on student learning in a web based instructional 
format. The use of training in standardized cooperative learning environments has been well established by 
other researchers. This study compared individualized network-based instruction; collaborative network- 
based instruction and cooperative network-based instruction to determine which of the two experimental 
groups came achieved statistical significance. This study examined several important educational theories 
that have contributed to understanding of the uses of cooperative learning in the classroom, and how 
theories of cooperative learning were used in relation to web based instruction. To accomplish this, the 
theory and literature of cooperative learning were reviewed. 

Literature Review 

Researchers involved in cooperative learning have looked extensively at research questions 
involving aspects of cooperative learning with and without computers. Questions involving group process, 
member ability, gender and race have all been examined to the satisfaction of many researchers. Issues 
involving training in cooperative learning skills, however, while extensively examined by those involved in 
the general field of cooperative learning research, have been neglected by researchers interested in 
cooperative computer-based instruction. 

Social Interdependence Theory 

Social interdependence exists when individuals share similar purposes for a task and success is 
also shared and reliant on other's actions (Johnson & Johnson, 1996). Deutsch’s (1949b) theory of 
cooperation and competition provided a focal point for the formulation of social interdependence theories 
in the mid- to late 1900s. Johnson and Johnson stated that Deutsch’s theory of cooperation and competition, 
established on the work of Lewin, was the focal point of future studies of cooperative learning (1991), 

Lewin 

Johnson and Johnson (1996) credit social interdependence theory’s origin to Kurt Lewin and the 
Gestalt school of psychology. Gestalt psychology is the study of how people view and comprehend the 
relation of the whole to the parts that make up that whole (Winn & Snyder, 1 996, p. 1 13), The influence of 
the Gestalt school of psychology on the use of cooperative learning comes from its influence on group 
structure. It is no longer necessary, as Lewin (1947) claimed it once was, for researchers to argue the very 
existence of a group within a societal structure. 

Deutsch 

Deutsch (1949a; 1949b) writing in the 1940s extended Lewin’s theories and formed a theory of 
cooperation and competition. Deutsch recognizes three types of social interdependence: positive, negative, 
and the absence of social interdependence. Under Deutsch's view, the type of interdependence within a 
situation determines an individual’s interactions, which subsequently determines outcomes (Johnson & 
Johnson, 1996). 

Positive interdependence is seen as promoting interaction between group members. Negative 
interdependence is seen as detracting group interaction. The condition of no interdependence results in an 
absence of group interaction. The idea of interdependence between group members has proceeded 
throughout the group literature to influence most of those investigating questions centered on cooperative 
learning, most notably D. Johnson and R. Johnson as well as Slavin. 

Cooperative Learning Theory 

Three of the most prominent researchers involved in researching cooperative learning are Slavin 
and D. Johnson and R. Johnson. Analyzing the work of the two Johnsons and Slavin lends a unique 
perspective to formal, structured cooperative learning. Johnson and Johnson (1996) view cooperative 
learning as the pedagogical use of small groups of two or more students so they work together to maximize 
their own and each other’s learning. Slavin (1994) believes all cooperative learning methods have certain 
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shared central ideas. In cooperative learning, students work together to learn and are responsible for one 
another’s learning as well as individual learning. 

Johnson and Johnson state there are four types of cooperative learning. In formal cooperative 
learning students work together to achieve shared learning goals and jointly complete educational tasks 
(Johnson & Johnson, 1996, p. 1018). In formal cooperative learning, teachers should tell the students the 
objectives for the lesson, make several preinstructional planning decisions, explain the task clearly to the 
students along with the need for positive interdependence, monitor student learning, intervene to provide 
assistance, and finally evaluate students’ learning and help the students evaluate their own learning. 

Informal cooperative learning is the second type of cooperative learning in which teachers assign 
students to work together to achieve a joint learning goal. The learning groups are temporary and meet for 
small periods of time (Johnson & Johnson, 1996). 

The third type of cooperative learning is the cooperative base group. Cooperative base groups 
consist of stable membership over a long time period with group members of mixed ability (Johnson & 
Johnson, 1996). Base groups are established to support peer group members throughout the academic year 
in the goals of making academic progress along with positive cognitive and social development. 

The fourth type of cooperative learning is academic controversy. Academic controversy exists 
when one student’s thoughts, ideas, or other knowledge formulations are incompatible with those of a 
second student and the two seek to reach an agreement (Johnson & Johnson, 1996). 

There are other significant cooperative learning methods such as Student Teams- Achievement 
Divisions (STAD), Teams-Games-Toumament (TGT), Team Assisted Individualization (TAI), Jigsaw, and 
group investigation. STAD consists of five major components (Slavin, 1994). These parts are class 
presentations, team activities, quizzes, individual improvement evaluations, and team recognition. TGT is 
similar to the STAD approach of cooperative learning (Slavin, 1994) with one significant instructional 
difference. TGT uses academic tournaments instead of STAD’s use of quizzes and individual improvement 
scores. TAI is focused on the individualization of mathematics instruction and uses a specialized set of 
curriculum materials (Slavin, 1 994). Jigsaw (Slavin, 1 994) is a form of instruction using cooperative 
learning techniques and is most appropriate when curriculum material is in written narrative form. Students 
working in Jigsaw, STAD, TAI, and TGT all work in heterogeneous teams. Group investigation is a form 
of cooperative learning that dates back to John Dewey (1974). Student success using group investigation 
requires prior training in communication and social skills. Group investigation is good for integrated 
educational projects dealing with the acquisition, analysis, and synthesis, of information to solve a complex 
problem. 

Cooperative Learning With Computers 

Webb (1987) reviewed the literature concerning peer interaction and learning with computers in 
groups. The goals of the review were to determine: (a) the pros and cons of group work for learning, (b) 
the types of verbal interactions that occur when small groups of students work at a computer, and (c) the 
types of interactions that are beneficial or detrimental to learning. She concluded that group work with 
computers was a feasible and capable way to learn. According to Webb, it was possible to design group 
learning settings that benefit most students (Webb, 1987). Hooper and Hannafin, Rysavy and Sales, and 
Simpson also published literature reviews or discussions of cooperative learning and computer-based 
instruction. All three articles found numerous positive influences of computers and cooperative learning. 
Adams and Anglin (Adams & Anglin, 1999; Anglin & Adams, 1996; Anglin & Adams, 1997) produced 
comprehensive reviews of more than 100 articles examining aspects of computers and cooperative learning. 

Computers and Cooperative Learning: Individual and Group Variables 

In their writings on the processes of computer assisted cooperative learning, D. Johnson and R. 
Johnson (1996) identify several key functions of group learning. These are positive interdependence, co- 
construction of ideas, conflict and controversy, giving and receiving help, equality of participation, 
promotive interaction, and division of labor. Positive interdependence (Johnson & Johnson, 1996) exists 
within a group when all members of a group believe that success is a joint endeavor. Group members are 
required to work together to achieve mutual, not individual success. Equality of participation (Johnson, 
Johnson, & Holubec, 1993) means that members of the group and the group as a whole are accountable. 
Group accountability exists when group performance is measured according to a particular criterion. 
Individual accountability is determined when the performance of individuals is assessed, and results of that 
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assessment are returned to the group for comparison against a measure of performance. Group members are 
held responsible for contributing an equitable share to the success of the group. Promotive interaction 
(Johnson et al., 1993) occurs when members of a group encourage and facilitate other members' tasks in 
order to reach the goals of the group. Giving and receiving help (Johnson et al., 1 993) occurs as group 
members aid each other in the pursuit of their mutual task. Division of labor (Johnson & Johnson, 1991) is 
the process of dividing the group task into manageable sub-tasks for group members. Conflict and 
controversy are also a part of the cooperative process. They occur as group members negotiate over the 
group task. Co-construction of ideas occurs as the learning occurs. Group members pursue a task and create 
a jointly constructed idea of their response to the instructional task. 

Scardamalia, Bereiter, Mclean, Swallow and Woodruff (1989) examined several instances of Joint 
construction of ideas in their investigation of computer supported intentional learning environments 
(CSILE). Students who were using a CSILE were provided a means to build a collective networked 
database examining a particular educational idea. Scardamalia et al. (1989) maintained that the use of 
CSILE networked environments aided in student construction of ideas through the use of direct 
contributions by students, and preparation for class contributions. Students were aided in the acquisition of 
higher order executive control of the learning process. The CSILE project was one of the most extensively 
researched projects (Oshima, 1989; Oshima, Bereitner, & Scardamalia, 1995; Oshima, Scardamalia, & 
Bereitner, 1996; Scardamalia, 1989; Scardamalia, Bereiter, & Lamon, 1994; Scardamalia & Bereitner, 

1991; Scardamalia & Bereitner, 1996; Scardamalia etal., 1992; Scardamalia et al., 1989) in cooperative 
computer-based instruction in the recent decade. The CSILE project involves the use of a networked 
database of notes and works created and shared via asynchronous communication. 

Studies organized by D. Johnson and R. Johnson were extremely influential in the investigation of 
cooperative learning strategies. In an article summarizing several pieces of research, they stated (1993) that 
cooperative learning promoted greater levels of oral discussion of curriculum material, higher achievement, 
and frequent use of higher order reasoning strategies when compared with competitive and individualistic 
learning. 

Group Processes 

Numerous researchers have examined different aspects of group processes, both in the traditional 
study of cooperative learning situations and in the newer study of computer-based cooperative learning. 
Webb and Lewis (1988) examined several aspects of help giving behavior during their program of research. 
Several factors of student discussions reflected positive correlations. These included giving explanations, 
and input suggestions, along with receiving responses to questions and receiving input suggestions in a 
group programming exercise. 

Hooper (1992) extensively researched behaviors surrounding the cooperative behaviors of giving 
and receiving help while using a computer between higher and lower ability students. Hooper (1992) found 
that high-ability students generated and received significantly more help in groups of similar ability levels 
than when placed in groups of mixed ability levels. Hooper's findings also found when grouped 
heterogeneously, high-ability students received lower amounts of stimulation in conversation from lower- 
ability students. Sherman (1994) examined help giving behavior. He found dyads using a cued HyperCard 
version of a treatment showed significantly more helping behaviors than those who used a noncued version 
of the same treatment. Several researchers noted the use of prompting cues in programs has potentially 
beneficial effects in increasing helping behaviors. 

Although several researchers examined behaviors surrounding the giving and receiving of help in 
cooperative computer use, a need for research still exists in this area. In a summary of past research and call 
for future studies. Light & Blaye (1 990) emphasized the need for researchers to be able to specify the ways 
in which children solicit and gain help from those around them while working at the computer. This 
knowledge of socially mediated help should be used to design help systems for computers. The primary 
research focus observed in the literature on computerized cooperative learning has centered on helping 
behaviors. Equality of participation, social loafing, and division of labor are all areas neglected in the 
literature. 

Training in Cooperative Learning Skills 

Johnson, Johnson and Holubec (1993) describe three types of cooperative learning skills. The first 
is described as functioning skills. These functioning skills are needed to manage the group effort for task 
completion and to maintain effective interpersonal relationships. The second set of skills is called 
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formulating skills. Formulating skills provide the mental processes needed to build increased levels of 
understanding of the material being studied, to stimulate the use of higher quality reasoning strategies, and 
to maximize mastery and retention of the assigned material. The third set of skills is fermenting skills. 
Fermenting skills involve the skills needed to participate in academic controversy. 

Relatively few researchers have examined the use of training in cooperative learning skills as an 
experimental variable. Several authors have used some form of computer-aided instruction to train 
participants, but this training did not extend to the use of cooperative learning techniques. Hooper, 
Temiyakam, and Williams (1993) trained students on the use of cooperative learning techniques, but did 
not measure the effects experimentally. Malouf, Wizer, Pilato and Grogan (1990) investigated the effect of 
training in cooperative learning techniques as an aid for special education students. Researchers 
documented an increase in cooperative learning interactions -the desired experimental effect. 

Repman (1993) compared three conditions in her experiments. Subjects were randomly placed in 
an unstructured setting, a structured setting, and a structured setting with training. Placing students in 
cooperative computer-based learning groups resulted in increased achievement in the content area. No 
differences were discovered regarding measures of critical thinking. 

Planning Instruction 

Several authors have examined the intricacies involved in planning instruction. Gagne' (1965; 
1984; 1978; 1992) has researched extensively the ways people learn and the conditions required for 
effective instruction. Dick and Carey (1990) have also extensively researched and written about the 
effectiveness of instruction. No study appears in the literature examining the use of cooperative learning 
when planning instruction. 

Kentucky Teacher Internship Program 

Phone interviews with officials from the Kentucky Teacher Internship Program, the Council of 
Chief State School Officers, and other leading educational organizations all revealed no known research on 
which their concept of lesson planning was based. One study examined first year teachers’ ability to assess 
students under the KTIP system. Sultana (1998) examined intern teachers’ ability to assess student 
performance on lesson objectives. Coders noted appropriate use of objectives in twenty-three to sixty-eight 
percent of assessment plans. A range of one-third to three-quarters of first year teachers was reported as not 
assessing instructional objectives in an appropriate manner. Lesson plan construction is mentioned in 
Guiding and Assessing Teacher Effectiveness (Certification, 1 998). However there is no guidance given 
teacher interns on methods of assessing those lesson plans within that document. It can be concluded that 
there is a need for further research in the training of how to assess lesson plans according to the KTIP 
system. 



Methodology 

Appropriate methods for investigating cooperative learning and computer use have ranged from 
the qualitative to the quantitative. Some researchers have undertaken a classical experimentalist 
investigation in regard to their research questions. Other researchers, in investigating the conversations 
between members of a cooperative group, have undertaken a qualitative analysis of the conversations and 
social interaction occurring within a cooperative learning experience. The particular method chosen for this 
research is a quasi-experiment. As such, it is appropriate to begin this discussion with an elaboration of the 
relevant variables. 

Independent Variables 

The independent variable in this study was instructional treatment. Three treatments were 
constructed to compare the effect of the absence or presence of training in cooperative learning strategies 
on individual learners. One treatment was labeled individual. All participants in the individual treatment 
received their instruction individually and were not allowed to work with classmates. The second treatment 
was labeled collaborative. All participants in the collaborative treatment were instructed to work together 
on the assignment and insure their partner understood the instructional content. The third treatment was 
labeled cooperative. All participants in the cooperative treatment were trained in cooperative learning 
techniques based on procedures outlined by Johnson, Johnson & Holubec (1993). 
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Dependent Variable 

The dependent variable used in this study was the evaluation score achieved by each subject as 
he/she participated in the experimental treatment. The evaluation tool was based on the Kentucky Teacher 
Internship Programs criteria for lesson plans and was constructed as a series of World Wide Web pages. 
Beta tests and revisions of the evaluation instrument and procedures were completed in the semester prior 
to the administration of the main study. 



Hypotheses 

The literature review suggested the need to investigate the interaction of cooperative learning with 
computer-based instruction. The following experimental question was generated from the review of 
literature: Given a computer to deliver lesson content and two teacher education students as learners, what 
is the most effective learning technique - individual, collaborative or cooperative? 

To illumine the experimental question, the following hypotheses were advanced. 

Hypothesis 1 . Using cooperative learning techniques with trained teacher-education student dyads 
using computer-based instruction, individual student instructional planning learning gains will be 
statistically significant when compared to students who received the instruction in an individualized setting. 

Hypothesis 2. Teacher-education students who receive training in cooperative learning techniques 
will produce statistically significant instructional planning learning gains than will students who received 
no training. 

Administration of the Treatments 

Teacher-education students enrolled in one of four sections of the same instructional technology 
class were the participants in the study. Two sections were each assigned an experimental treatment, one 
collaborative, the other cooperative. Two other classes were combined to form the third experimental 
treatment group, the individual. Administering the treatment according to this design allowed all students 
within the same class to receive experimental treatments the same way at the same time. Previous studies 
have noted the negative effects of different students receiving different treatments at the same time. 
Administering treatments to all students within a classroom grouping at the same time reduced the 
possibility of subject discomfort. 

Two assessments were given to all students. The initial instructional assessment checked the 
students' ability to evaluate lesson plans according to the Kentucky Teacher Internship Program standards. 
The Internship Program standards are extrapolated from the requirements established in earlier documents 
cited within the literature review. The use of the term “standards” to describe lesson plan evaluation is not 
meant to imply a national group charged with setting evaluative guidelines has constructed these tools. The 
second delayed assessment measured the students' knowledge of lesson plan construction during the 
administration of the web-based treatment. 

Participants 

The fifty-four participants for this quasi-experiment were college students enrolled in an 
undergraduate instructional technology course for preservice teachers. The students were sophomores or 
Juniors who had been admitted to a teacher education program at a large southern university. Two intact 
groups of students were administered the treatments (collaborative and cooperative). The group of 
individual participants served as a control function and received the individual treatment. 

Materials 

Instructional content for all groups was delivered via a World Wide Web page. This instruction 
was based upon the standards developed for the Kentucky Teacher Internship Program (KTIP). The web 
site developed for this experiment instructed students in the evaluation of lesson plans according to the 
KTIP lesson plan standards outlined in Guiding and Assessing Teacher Effectiveness (Certification, 1998). 
A web-based evaluation was administered following completion of the experimental treatment. A second 
assessment was administered a week later. A copy of this web site is available on request from the author. 



BEST COPY AVAILABLE 

18 



Instrument 

The experimental instrument measures students’ ability to evaluate lesson plans according to 
Kentucky Teacher Internship Program standards. The way the instrument measures these concepts is by a 
series of Likert type questions. Each question was based on an individual evaluation statement issued in 
Guiding and Assessing Teacher Effectiveness (Certification, 1998). The instrument was developed over a 
six-month period. The development process consisted of creation and multiple revisions of questions, 
consultation with Internship graduates and officials, and conducting beta testing and revision of the 
questions. The beta testing consisted of administering the questions to students from the same population as 
the experimental population. Following the initial beta testing, discussions were held with students to 
determine if they understood the questions. A reliability analysis was conducted on evaluation questions 
used by all subjects. Cronbach's alpha measuring internal consistency of item evaluations was found to be 
a = .7209. The standardized item alpha was computed as a = .8631 . Results from the reliability analysis 
are listed in Table 5. If the individual question was a yes/no question, the possible answer for that question 
was coded as either one or two respectively. In other questions evaluation scores were coded as Likert type 
response ranging from one to five. A score of one was the lowest possible score for that item. A score of 
five was the largest possible score for that item. The same measure was used for both Task 1 and Task 2. 
The lessons were different but of comparable quality. A delay of one week occurred between the two 
sessions. The one interim class meeting dealt with basic computer skills. No content dealing with the 
research was discussed during that class meeting. 

Design 

This study examined the effects of individual, cooperative and collaborative learning 
environments on teacher education student learning of instructional planning in a web based instructional 
format. The experimental design is shown in Figure 2. 

Table I: Desi2n of the Study 



Groups 





Individual 


Collaborative 


Cooperative 


Time 1 


N=19 


N=6 


N=28 


Time 2 


N=18 


N=7 


N=20 



The individual column was a control treatment. Students assigned to this treatment completed the 
lesson plan analysis alone. 

Participants assigned to column two were placed in the collaborative learning condition. 
Collaborative students completed the experimental treatment in dyads and were told as a part of the class 
assignment they should work together while completing the assigned task. These dyad members were only 
told they should attempt to ensure their partners understood the web based instructional content. Subjects 
received no guidance from the instructor on methods for working together. 

Students in the cooperative learning experimental group were trained in cooperative learning 
skills. As some authors have noted (Carrier & Sales, 1987; Eraut & Hoyles, 1989; Hooper et al., 1993; 
Repman, 1993), training in cooperative learning skills was ignored in the literature concerning impact of 
cooperative learning. Not enough research has been conducted about the impact of training in cooperative 
learning skills, especially in a computer-oriented condition. 

Students in this third group were trained in three types of cooperative learning skills based on 
techniques outlined by Johnson, Johnson and Holubec (1993). The first was described as functioning skills. 
These functioning skills are needed to manage the group effort for task completion and to maintain 
effective interpersonal relationships. The second set of skills is called formulating skills. Formulating skills 
provided the mental processes needed to build increased levels of understanding of the material being 
studied, to stimulate the use of higher quality reasoning strategies, and to maximize mastery and retention 
of the assigned material (Johnson, Johnson and Holubec; 1993). The third set of skills is fermenting skills. 
Fermenting skills involved the skills needed to participate in academic controversy. 

These three sets of cooperative learning skills were presented to students in the cooperative 
learning condition using a PowerPoint presentation. Students were asked to practice these skills briefly 
during the class period in order to become more experienced cooperative learners. Following the training, 
student dyads reviewed the same web based treatment as the students in the other two groups. 
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Data Analysis 

The data were analyzed using SPSS for Windows. Primary hypotheses were tested using the 
Univariate Analysis of Variance grouping of procedures. The following linear regression model was 
examined: SUMMARY = CONDITION * LEVEL (CONDITION*LEVEL). Statistically significant 
within- subjects effects for performance score means were found to exist following the treatment. The 
results of all analyses are reported below beginning with multiple descriptive statistics. 

Descriptive Statistics 

The first collection of data to be represented in this dissertation is group mean performance scores 

on items. 

Means by group 

Mean scores for posttest evaluations of the lesson plans by each grouping condition are listed in 
Table 2. The mean evaluation score for subjects in the individual group was 95. 1 8 (SD = 13.46) and was 
100.62 (SD = 12.03) for those in the collaborative group. Evaluation mean score for subjects in the 
cooperative group was 101.85 (SD = 15.55). 

There were some obvious changes in reported evaluation scores between Task 1 and Task 2. These 
scores are reported in Table 2. The mean evaluation score for the individual group in Task 1 was 92.95. 

The Task 2 mean evaluation score was 97.56. In the collaborative group during Task 1 , the evaluation 
mean score was 109.71 . During the second evaluation, the mean evaluation score was = 90.00. Subjects in 
the cooperative group during Task 1 had a mean evaluation score of 102.07. The cooperative Task 2 mean 
evaluation score was 101 .55. 

Table 2: Comparison Table of Means by Task 



reatment 


Task 1 
mean 
score 


Task 1 
standard 
deviation 


Task 2 
mean 
score 


Task 2 
standard 
deviation 


Task 1 and 2 
pooled 
mean score 


Individual 


92.95 


.39 


97.56 


.40 


95.19 


Collaborative 


109.71 


.26 


90.00 


.24 


100.62 


Cooperative 


102.07 


.45 


101.55 


.39 


101.85 



Reliability Analysis 

Reliability analysis was conducted on the evaluation scale items used by all participants. The 
concept of reliability refers to how accurate, within the means of statistical controls; an estimate of the true 
score is within a measured population. Tests evaluating Cronbach's alpha measuring internal consistency of 
item evaluations were conducted. The proportion of variability in instrument responses that was a result of 
respondent differences was rated as a =. 8010. This procedure describes and represents the degree to which 
scores on an evaluation depict something other than measurement error. Results from the reliability 
analysis are listed in Table 3. 

Table 3: Analysis o f Variance for Criterion Measure 

Source of Sum of Squares DF Mean Square F Probability 

Variation 



Between 604.8315 97 6.2354 

People 



8 



18 



Within 

People 


5902.8000 


3332 


1.7715 


Between 

Measures 


1811.3866 


34 


53.2761 


Residual 


4091.4134 


3298 


1.2406 


Total 


6507.6315 


3429 


1.8978 


Grand Mean 


2.9353 






Reliability 

coefficients 


34 Items 


98 Cases 




Alpha 


.8010 







42.9447 .00 



Hypotheses Tests 

A Univariate linear regression model repeated measures analysis was conducted. The general 
linear model regression was selected because of its ability to provide both regression analysis and analysis 
of variance within groups. Significance was found to exist for the variable of grouping. The grouping 
variable consisted of the combined individual, collaborative and cooperative experimental condition 
groups. The grouping variable was constructed to allow for testing to evaluate if any significant results 
were found within any of the three experimental conditions. Students who participated in each level of the 
experiment completed one evaluation of a lesson presented on a web site. One week later, the same 
students in the same conditions evaluated a second lesson. The following linear regression model was 
proposed to examine experimental effects. 

Summary=Condition * Level * (Condition*Level) 

A complete result of Univariate between-subjects effects is reported in Table 4. Similar multiple 
comparisons are reported in Table 5. 
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Table 4: Univariate between-subiects effects 



Source 


Type III Sum of Squares 


Df 


Mean Square 


F 


Sig. 


Corrected Model 


2414.423(a) 


5 


482.885 


2.444 


.040 


Intercept 


700489.978 


1 


700489.978 


3546.077 


.000 


Condition 


897.556 


2 


448.778 


2.272 


.109 


Level 


485.116 


1 


485.116 


2.456 


.121 


Condition * Level 


1424.500 


2 


712.250 


3.606 


.031 


Error 


18179.628 


92 


197.539 






Total 


984455.000 


98 








Corrected Total 


20588.051 


97 









(a) = . 1 17 (Adjusted =.069) 



Earlier analysis showed a significant effect to exist within the combined statistical variable of 
grouping. Further analysis was necessary to determine which of the variables contained the greatest 
significance. A pairwise analysis of variance for all grouping conditions using the Least Significant 
Differences method showed a significant difference between the individual and cooperative conditions (M= 
6.6650, SE = 3.0748, p=. 033). These statistics are reported in Table 5. 

Table 5: Multiple Comparisons 



(I) 

Condition 


(j) 

Condition 


Mean 

Difference 

(i-j) 


Standard 

Error 


Sig. 


95 % Confidence Level 
Lower Upper 

Bound Bound 


Individual 


Collaborative 


-5.4262 


4.5315 


.234 


-14.4261 


3.5737 




Cooperative 


*^-6.6650 


3.0748 


.033 


-12.7718 


-0.5582 


Collaborative 


Individual 


5.4262 


4.5315 


.234 


-3.5737 


14.4261 




Cooperative 


-1.2388 


4.3944 


.779 


-9.9664 


7.4889 


Cooperative 


Individual 


♦6.6650 


3.0748 


.033 


0.5582 


12.7718 




Collaborative 


1.2388 


4.3944 


.779 


-7.4889 


9.9664 



The mean difference is significant at the .05 level 



Figure 3 provided a visual comparison of pooled Task 1 and Task 2 performance score means of 
each experimental condition. Several observations were drawn from this comparison. Subjects in the 
collaborative condition experienced a dramatic drop between the two times. Individual learners experienced 
a gain in scores between times. Cooperative subjects were the most consistent of any of the experimental 
treatments. 
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Figure 3: Comparison of Means for Individual, Collaborative and Cooperative Conditions by Task 1 and 2, 






Task 1 


Task 2 


-♦—Individual 


92.95 


;V: 97.56 


Collaborative 


109.71 




— Cooperative 


102.07 
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Discussion, Conclusions and Summary 

The goal of this research was to determine the best way to structure learning involving two teacher 
education students at a computer. Summary of research findings, conclusions, limitations, and 
recommendations for future use will be presented. 

Summary of Findings 

This study revealed several interesting results. Based on the statistical analysis, significant 
differences were found between the three treatments. Data are examined by hypothesis in sections below. 

Hypothesis I 

Hypothesis 1 proposed that when teaching cooperative learning techniques with student dyads 
using computer-based instruction and trained in cooperative learning techniques, individual student 
learning gains would be statistically significant compared to than those students who received the 
computer-based instruction in an individualized setting. The data provided by this research suggest that 
statistically significant differences exist between students in the cooperative treatment and students in the 
individual control conditions. 

Hypothesis 2 

Hypothesis 2 suggested teacher-education students who receive training in cooperative learning 
techniques would produce statistically significant instructional planning learning gains than will students 
who received no training. Differences were established between condition means, however, these 
differences were not statistically significant. 

Students in the cooperative experimental condition produced more consistent scores across time. 
These students in the cooperative learning condition then produced higher learning gains than students with 
no training. 

Collaborative learners showed the largest point drop between the two lessons. During the lesson 
used in Task 1, collaborative learners ranked the first lesson higher than all participants. The collaborative 
group, however, also ranked the second lesson twenty points lower than the first. None of the other groups 
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exhibited such a change in evaluation score. All groups other than the individual exhibited a negative point 
change in comparison from Task 1 to Task 2. From the evidence in this research, the collaborative group 
appeared to produce a large negative effect on subject learning. 

Few apparent reasons indicated such a change. Reasons for such a point drop were drawn from the 
description of experimental conditions themselves. Individual participants were only provided with the web 
based instructional resource. Collaborative participants were provided v^nth a peer to discuss the instruction 
as it was being administered. Cooperative learners were also provided with a peer for discussions, and in 
addition, were also given specific training and strategies for learning together in a cooperative mode. One 
possible conclusion drawn from the data is the lack of structure in the collaborative mode could have been 
a contributing factor for the change in grand mean scores between Task 1 and Task 2. Lack of structure 
may have affected Task 2 more than Task 1 because possibly students were more critical of the 
instructional measure. A larger group of subjects in the collaborative condition also could have influenced 
the experimental result. 

The scores for students in the individual group rose from Task 1 to Task 2 nearly five points for 
the overall summary score. This was the only group to experience a rise in point values of the lessons 
evaluated. The rise in scores could have been attributed to a practice effect. Students may have become 
more skilled in the use of the measure over time. Questions then arise about why a practice effect was not 
visible in the other groups. The major factor that could impede the use of an individual practice effect was 
the inclusion of a peer for communication purposes. Piaget and Vygotsky documented the effectiveness of 
peer communication as a learning aid. Mixed results happened in this research. 

What is the most effective way to provide structured instruction for two learners at a computer? 
The results indicate cooperative learning is the most effective way to structure computer-based instruction 
with two students. The significant effect of grouping using the LSD analysis for the cooperative learning 
treatment establishes the cooperative treatment as the most effective. 

A statistically significant condition also exists between the individual and experimental 
conditions (or treatments). Evidence of statistical significance between the individual and collaborative 
treatment and the collaborative and cooperative treatments exists 

Conclusions 

This study has several significant conclusions. Of the three methods of computer based instruction 
- individual, collaborative, and cooperative - cooperative computer based instruction appears to be the 
most effective. More research is definitely called for in the area of lesson plan evaluation and Kentucky 
Teacher Internship. 

Recommendations for Future Research 

This research leads to several questions for future research. The use of training in cooperative 
learning strategies during computer-based instruction should be investigated further. The effect of training 
on cooperative learners has been established as an intriguing possibility. 

Why was the drop in grand mean scores so extreme for collaborative learners? Can this be 
duplicated in other instructional settings? In Task 1, the collaborative learners have the highest scores of 
any group involved in the research, thus what factor produces the drop in grand mean scores? Investigation 
of verbal and nonverbal communications occurring between group members may provide answers to 
researchers examining similar questions. 

One of the foci of this research was to examine cooperation in a computerized environment. As 
such, a natural starting place was to examine the cooperative environment between two or more people as 
they learned together at the computer. Research outlined in this study identified preferable ways to 
structure learning when learners share the same computer. Future research should examine the different 
ways people and computers interact in a learning situation. With the growth of instruction in web-based 
settings, researchers should examine the interaction between students mediated by a web based 
instructional tool. This study has examined students’ learning and interaction when sitting in front of the 
same computer. Future researchers should examine the related questions of how students learn and interact 
when working through a computer mediated environment. 
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Recommendations for Instruction 

Teachers have often confronted the problem of group learning. In recent decades, instructional 
designers have been forming ideas about the structure of cooperative learning v^hile at the computer. This 
study compared three ways of structuring learning: individual learning at the computer, collaborative 
learning at the computer between pairs and cooperative learning at the computer. 

Individual Learning at the Computer 

This research established that individual learning at the computer is a somewhat effective but not 
statistically significant mode of learning for students. Instructors who structure learning events for 
individual students at the computer should let them practice these events over time. Research detailed 
earlier in this paper shows a slight practice effect for scores over time. As an instructional strategy, the use 
of individualized computer-based instruction does not appear to be the preferred strategy instructors should 
use. 

Collaborative Learning at the Computer 

Collaborative computer-based instruction appears to be the most unreliable of any of the 
instructional strategies compared in this research. Collaborative learners produced the highest score of any 
of the experimental treatments examined. However, the scores have the largest change in any of the 
treatments examined. The major perceived reason for that change is the lack of structure regarding 
collaboration in the learning environment. Collaborative learning appears to be an effective learning 
strategy for short term learning events but not for events over longer periods of time. As such, the use of 
collaborative learning in computer-based instructional situations should be limited. In short term learning 
events collaborative computer-based instruction appears to be an acceptable mode of instruction. For 
learning events that occur over a longer period of time, the use of collaborative learning as an instructional 
strategy should be seriously questioned. If these results are duplicated in other research, instructors should 
seriously examine their use of collaborative learning techniques at the computer. 

Cooperative Learning at the Computer 

Given the stated topic and audience, cooperative computer-based instruction emerged from this 
experimental comparison as the preferable mode of instruction. Students in the cooperative learning mode 
had the highest evaluation scores of any of the experimental treatments. Cooperative learning scores were 
the most consistent of all the groups. Scores were higher than the individual group, but lower than the 
collaborative group during Time 1 of the experiment. However, students in the cooperative grouping 
maintained a consistency of scoring over the second time in the experiment. Based on the research detailed 
in earlier sections, educators who are involving students in computer-based instruction in situations with 
two students at the computer should train those students in effective means of cooperative learning. 

Limitations 

This research has several limitations. One primary limitation for this study was the population of 
sophomore and junior education majors. This limitation could be addressed by increasing the size to 
include more types of people and more varied age groups of people. 

A second limitation was the length of the treatment. Students were given the initial version of the 
treatment and tested regarding knowledge gains within a week. This compressed time span was partly 
attributable to the nature of the class in which the subjects were recruited. The class lasted for a typical 
period of one month, with twelve class meetings. Researchers examining these experimental conditions 
under differing time spans may find differing results. 

A third limitation of this research was the sample size. The result of this study would have been 
improved with the use of equal sample sizes between the individual, collaborative and cooperative 
conditions. This research had intact classes performing the same treatment. The selection of intact classes 
led to the use of unequal cell sizes. Experimental power would be improved in future research with 
identical cell sizes. 

A fourth limitation was the recruitment of a convenience sample. The students involved in this 
quasi-experimental research were recruited as a convenience sample from courses partially taught by the 
author. 
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A final limitation was the use of intact classes to determine sample size. Intact classes were used 
to determine sample size because this experimental treatment was used as an instructional treatment. 
Students would raise questions if given differing instructional treatments in view of each other. If sample 
size were determined strictly by statistical requirements for experimental power, this limitation would be 
overcome. 

Summary and Discussion 

\This study compared a control condition of individual teacher education students at the computer 
with experimental conditions of collaborative teacher education students at the computer, and trained 
teacher education students engaging in cooperative learning at the computer on two tasks administered one 
week apart. The tasks were to evaluate two different lesson plans using one evaluation measure. All 
experimental groups were compared to a control condition for the evaluation of lesson plans. Fifty-four 
subjects participated in the study. 

The experimental treatments were delivered in two parts. One part remained identical across all 
groups. This web-based portion of the experimental treatment teaches students to evaluate lesson plans 
according to guidelines of the Kentucky Teacher Internship Program. The second part required one group 
of students to complete the web based instruction individually, another group to work in collaborative 
dyads in which they were given no guidance from instructors on working together, and a third to work in a 
cooperative learning mode after being trained in cooperative learning skills. All groups were asked to 
evaluate the same two lesson plans using identically formatted lesson plans. 

Statistically significant differences were found. The effect of grouping students was found to be 
especially notable. Significant comparisons were made between subjects in the cooperative group and the 
individual group. 

What is the most effective means to structure students' learning while at the computer? Results 
from this research suggest instructors should place students working together in a cooperative situation and 
that these students should be trained in cooperative learning techniques. The comparison between 
individual and cooperative learners proves to be a major significant comparison of this experiment. 

Should students be placed together at the computer? During this research, when students were 
paired at the computer, subjects achieved higher scores. Depending on experimental treatment, significant 
amount of variance appeared in the evaluation scores. Cooperative learners produced higher, more 
consistent evaluations. This research sees no reason to avoid placing two students at the same computer. 

If scores are higher when two students are at the same computer, should they be taught how to 
work together? Students should be trained in cooperative group work skills, whether in cooperative or 
collaborative settings. The effectiveness of training in cooperative skills has been well established in 
studies of cooperative learning in general instruction. This research duplicates and extends that research 
into the area of computer-based instruction. 
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A Comparison Of Students’ And Teachers’ Computer 
Affect And Behavior Effects On Performance 



Roy M, Bohlin 
California State University 



Keller (1983) has presented a macro-model for the interaction of instruction with learner 
characteristics. He integrated the works of many other theorists into this model. Bohlin (1998) later 
developed a Model of Learner-Instruction Interactions in the Affective Domain which was built upon this 
and other models. This theory-based model included the interactive relationship of instructional, 
psychological, and behavioral factors. Because this model was theoretically based, there was a need to 
determine if empirical data could validate it to some extent. This data could be used to improve the 
accuracy of the model through formative evaluation. 

Tobias (1979) proposed a model for the effect of anxiety on learning from instruction. This model 
suggested that anxiety interferes most with learning before, and during input by the learner. Before 
processing, anxiety acts as a diversion to attention. During processing, anxiety directly interferes with the 
cognitive processing and storage of information by the learner. Postprocessing anxiety obstructs later 
retrieval of content mastered during instruction. Eysenck (1984) found that actual performance for high 
anxiety individuals and low anxiety individuals is less clear than the anxiety differences. Anxiety level, 
itself, is not the only determining factor on behavior and choice. 

Bohlin, Milheim, and Viechnicki (1993) found that age is an important variable in the motivation 
of learners. Therefore, it would be important to determine if the model might differ for K-12 students 
versus teachers. One of the critical times to assess the perspectives of students is during the time that they 
are making important decisions regarding their future careers. Eighth and ninth grade students were 
selected for the target population with this in mind. 

The purpose of this study, therefore, was to determine how the variables computer anxiety, 
confidence, attitudes, self-efficacy, expectations, motivation, effort, perceptions of abilities and knowledge, 
to performance on computers are related and to what degree they predict the computer performance of 
students. The following research questions were investigated: 

(a) What are the relationships of students' and of teachers’ performance on computers to 
computer anxiety, confidence, attitudes, self-efficacy, expectations, motivation, effort, 
perceptions of abilities and knowledge? 

(b) How do computer anxiety, confidence, attitudes, self-efficacy, expectations, 
motivation, effort, perceptions of abilities and knowledge predict students' and teachers’ 
performance on computers? 



Methods 

Loyd and Gressard's (1984) Computer Attitude Scale (comprised of four subscales measures the 
subjects’ anxiety, confidence, relevance and attitude levels) was administered to 97 in- and pre-service 
teachers enrolled in computer methods classes and to 137 students enrolled in 8*^ or 9*^ grade at three 
schools. The subjects were also asked to respond to a series of questions that were used to measure their 
levels regarding computer-related (a) motivation; (b) perceived ability to learn; (c) perceived knowledge; 
(d) effort; (e) expectations of their performance in the class; (f) expectations of the amount they will learn 
in the class; and (g) self-efficacy. 

Within one week a hands-on assessment of the subjects’ computer performance was administered. 
The data were used to calculate a correlation matrix, a stepwise regression equation to predict the 
dependent variable, and to perform path analyses for each set of data. The data were collected during the 
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approximate midpoint of the semester so that beliefs were most likely to based upon first hand expereiences 
with computers and therefore most likely to be as accurate as possible. 

Analysis & Results 

Teachers 

The factors that contributed the most in predicting the computer performance of the teachers was 
their confidence, liking of computers, perceptions of computer usefulness, self-efficacy toward computers, 
and amount of learning expected in the computer course (see below). Confidence and perceived usefulness 
scores were associated with higher performance scores for these teachers. 

Table 1. Regression Eauation for Teachers 

Performance = -3.486 + 0.41xj - 0.28x2 0.23x3 ~ i -28 X4 - 0.57 X5 p = .009 



Standardized Beta Coefficient 

Xi = Confidence toward computers (forty-point scale) +2.06 

X2 = Liking of computers (forty-point scale) -1.55 

X3= Usefulness of computers (forty-point scale) +0.73 

X4 = Self-efficacy toward computers (five-point scale) -0.87 

X5 = Expectations of learning in course (five-point scale) -0.39 



Note: Performance is measured on a five-point scale (A = 5; B = 4; C = 3; D = 2; High F - 1 ; Low F - 

0 ) 

Teachers' own reported knowledge and skills about computers was the first variable removed 
from the regression equation. Anxiety level and motivational level were the second and third variables 
removed. A Path Analysis of the teachers' data was performed using the theoretical model. The results of 
this analysis is represented in Figure 1 . 

Students 

The factors that contribute the most in predicting the computer performance of the teachers was 
their anxiety, ability in learning about computers, effort, and amount of learning expected in course (see 
below). Confidence and expectation scores were associated with higher performance scores for these 
students. 
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Figure 1 . A Computer Affect and Performance Model for Teachers 



Table 2. Regression Equation for Teachers 



Performance = 0.964 - 0.29xi + 1 .85x2 0.63x3 ■ 0*98 X 4 p = .0002 

Standardized Beta Coefficient 

Xi = Anxiety toward computers (forty-point scale) -0.57 

X 2 = Ability in learning about computers (five-point scale) +0.50 
X 3 = Effort (ten-point scale) +0.27 

X 4 = Expectations of learning in course (five-point scale) -0.29 



Note: Performance is measured on a five-point scale (A = 5;B = 4;C = 3;D~2; High F ~ 1 ; Low F ~ 0) 

The students’ own reported motivation level was the first variable removed from the regression 
equation. A Path Analysis of the teachers’ data was performed using the theoretical model. The results of 
this analysis is represented in Figure 2. 
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Figure 2 . A Computer Affect and Performance Model for Eighth and Ninth Grade Students 



Conclusions 

One surprising finding was the large number of negatively related factors to performance for both 
groups. Based upon theory and much previous experimental findings, it certainly would be expected that 
self-efficacy, effort, and attitudes are important to learners* achievement. 

It is interesting to note that the strongest positive factor in predicting higher achievement for 
teachers was confidence toward computers. Although confidence was highly negatively correlated to effort 
(-0.90), it still is an important factor when addressing degree of performance. It also was not surprising that 
motivation was most highly correlated to perceived usefulness of computers (0.83), since relevance is an 
important motivational factor. For students, the strongest predictor of achievement was level of anxiety. 
This supports the need for interventions to help reduce students* anxiety. 

Perhaps the biggest surprise was the fact that knowledge about computers was the first excluded 
variable from the regression equation for teachers. The fact that anxiety toward computers was the second 
excluded variable for teachers is not as surprising, because (unless it is extreme) sophisticated anxious 
learners are known to have the ability to compensate for their anxiety by increasing effort and reducing the 
debilitating effects of the anxiety. In this study, anxiety did have a correlation of 0.76 with effort. Anxiety’s 
greatest influence usually is on avoidance in voluntary choices that one makes. Some experimental 
evidence suggests that highly motivated high anxiety learners sometimes compensate for the effects of the 
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anxiety by increasing their effort on the task. This did seem to be the case with the student group, since 
higher anxiety correlated with better performance on the hands-on test. 

The order inwhich the factors contribute to the performance measure are approximately the same 
order as these variables are reverse sequenced on the Model from performance. Since motivation seems to 
interact so closely with perceptions of need by the learner, their effects are somewhat mixed and did not 
show up statistically in the results of the analyses. 

Further research 

Further research is suggested to gather enough data to perform path analyses to further validate the 
Anxiety, Confidence, Attitudes and Performance Model. Further research that investigates other types of 
anxiety (such as mathematics or test anxiety) can help provide insight into the differences between these 
different constructs. Studies that investigate different types of computer performance (e.g. WWW 
searching, multimedia development, word-processing) may provide insights into important differences in 
contributing factors. Another area of needed research is to gather data from adults involved in training 
sessions. 
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Effects of Headings and Computer Experience in CBI 



Jonathan D. Brinkerhoff, James Klein, and Carol M. Koroghlanian 
Arizona State University 



Abstract 

The purpose of the study was to investigate the effects of overview mode and computer experience 
in a hypermediated learning environment. Subjects read a hypermedia unit that included either a 
structured overview, unstructured overview or no overview. The study examined the effects of 
overview mode and computer experience on achievement, attitude and instructional time. Results 
indicated that participants with high computer experience learned more from the hypertext 
program than those with low computer experience. Furthermore, participants who received either 
the structured or unstructured overview spent significantly more time on the program and had 
significantly more positive attitudes than participants who did not receive an overview. However, 
overview mode did not influence achievement. 

The proliferation of Internet-based instruction suggests that learners will have increasing 
opportunities to use hypertext A major advantage of hypertext is its flexibility for accessing and linking 
topics, however, the advantages of flexible data access may be offset by difficulties with navigation and 
remaining oriented within the structure of the hypertext (McDonald & Stevenson, 1996; Kozma, 1991; 
Lawless, 1996; Jonassen, 1988). Disorientation in hypertext due to its poor organization represents a 
significant educational problem, as cognitive resources devoted to matters of navigation reduce the 
resources available for learning (Jonassen, 1988; Sweller, 1994; McDonald & Stevenson, 1996). 

One means for limiting the disorientation associated with hypertext involves the inclusion of some 
type of content overview to provide the hypertext with a level of structure (Jonassen, 1986). Research has 
examined a variety of forms of overviews including advance organizers, graphic organizers and headings. 

In a meta analysis of advance organizer research, Mayer (1979) was able to document an interaction 
between use of advance organizers and printed text organization. Advance organizers enhanced learning 
outcomes for poorly organized text or text presented in an unfamiliar form, while having no effect on 
learning from well organized text or text presented in a familiar form. Graphic organizers have also been 
shown to be effective. In a meta analysis examining the efficacy of instructional orienting activities in 
computer-based instruction (CBI), Kenny (1994) reported that the inclusion of graphic organizers could be 
effective in enhancing CBI. Headings represent another means for providing structure to poorly organized 
text by explicating the relationships between superordinate and subordinate topics. Various studies have 
reported positive effects of headings on recall with print materials (Lorch & Lorch, 1996; Hartley & 
Trueman, 1985; Wilhite, 1986; Hartley, Kenely, Owen, & Trueman, 1980; Holley, Dansereau, Evans, 
Collins, Brooks, & Larson, 1981). 

Overviews such as graphic organizers and headings can represent content topics in a variety of 
arrangements such as alphabetical or hierarchical. Researchers have compared the effectiveness of 
differing overview organizations. Brooks and Dansereau (1983) found recall of hierarchically formatted 
text superior to that of nonhierarchical text; Willerman and Harg (1991) reported higher levels of recall for 
subjects given a hierarchically organized concept map. Dee-Lucas and Larkin (1 995) compared the effects 
of a hierarchically organized interactive overview to an unstructured overview where topics were listed 
alphabetically and found recall levels higher with the hierarchical overview. The superiority of 
hierarchically structured overviews is thought to result when recall of superordinate nodes spurs recall of 
subordinate nodes (Kardash, Royer, & Greene 1988; Kintsch & Yarbrough, 1982; Lawless & Brown, 1997; 
Lorch & Lorch, 1996). 

These results suggest that the educational efficacy of hypertext may be enhanced by inclusion of 
some form of hierarchically organized orienting activity designed to reduce problems with navigation and 
orientation in hypertext. In selecting an appropriate orienting device, McDonald and Stevenson (1998) 
maintain that graphic organizers aren’t practical for larger databases while simultaneously commandeering 
an exorbitant amount of screen space. Such practical considerations suggest that hierarchically organized 
headings might represent an effective form of orienting activity in a hypertext environment. However, 
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while the cited studies support use of headings as orienting activities, they were conducted with print 
media. The degree to which those findings transfer to a hypertext setting remains unclear. 

Other factors such as computer experience, may also influence navigation and learning from 
hypertext. Learners more familiar with hypertext may expend fewer mental resources on navigation and 
orientation than learners less familiar with hypertext, leaving more of those resources available for 
knowledge assimilation. Swan, Bowman, Vargas, Schweig, and Holmes (1998) looked at how people 
make sense of electronically presented information on the World Wide Web and found that more 
experienced hypertext users navigated through electronic texts with greater ease than those with less 
experience. This implies that those more familiar with hypertext may require fewer cognitive resources 
focused on navigation, leaving greater resources available for assimilation of content. Beyond simple 
navigational expertise, increased computer experience may contribute to an increased ability to recognize 
the organizational structure of hypertext. In related studies, Ayersman and Reed (1998) and Reed, 
Ayersman and Liu (1996) examined subjects' ability to identify the underlying organizational structure of 
hypertext as either linear semantic networks / scripts, or nonlinear concept maps / schemata. Both reported 
that subjects with greater computer experience showed greater facility at recognizing hypertext structure, 
suggesting that those with more extensive hypermedia experience may be able to utilize nonlinear 
presentations more effectively. 

The purpose of the current study was to examine the effects of structured overviews and computer 
experience on learning from hypertext. More specifically, what are the effects of overview mode 
(structured, unstructured and none) and computer experience (high, low) on achievement, attitude and 
instructional time in a hypermediated learning environment. 



Method 

Design and Participants 

A 3 X 2 factorial design was used for this study, with overview mode (structured, unstructured and 
none) and computer experience (high versus low) as the independent variables. The dependent variables 
were achievement, attitude and time in program. 

Seventy-nine undergraduates (50 females, 29 males) enrolled in an instructional methods course at 
a large southwestern university participated in this study. The course, conducted through the College of 
Education, fulfilled program requirements for education majors while representing an elective course for 
the remaining participants. Seventy-seven percent of the subjects were education majors, while the 
remaining 23% spanned majors from accounting to real estate. While representing all undergraduate 
levels, subjects were predominantly juniors and seniors (71%). 

Materials 

Materials used in this study included a computer based instructional (CBI) program and a 
demographic survey. The CBI program consisted of three versions of Writing Objectives Using the 
Outcomes of Learning, an original, self-paced, hypermedia program covering concepts relating to the types 
of learned performances as described by Robert Gagne (Gagne and Driscoll, 1988). Topics presented 
included the five categories of learning outcomes: motor skills, attitude, verbal skills, intellectual skills and 
cognitive strategies. Subtopics comprised the five types of intellectual skills (discriminations, concrete 
concepts, defined concepts, rules and higher order rules) and the three types of verbal information (names / 
labels, facts and bodies of knowledge). These topics related directly to the instructional content of the 
course. 

The hypermedia program was developed in three versions representing the three treatment 
conditions (structured overview, unstructured overview, and no overview). All versions covered identical 
content: a description of the navigational aspects of the program, an introduction to the material delineating 
program content and the relationship between the outcomes of learning and the writing of effective 
instructional objectives, information concerning the five categories of learning outcomes, and action verbs 
suitable for instructional objectives reflecting each of the learning outcome categories. The structured 
overview and unstructured overview versions contained 18 introductory screens and 56 information 
screens, while the no overview version contained 14 introductory screens and 51 informational screens. 
Additionally, all versions of the CBI included a 22-item, untimed posttest. 
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Information was presented for each category of learning outcome through text and graphics. The 
CBI program permitted selection of topics in random order. Within topics, information presentation was 
linear, however subjects could opt to return to previous screens, switch to an alternate topic, or begin the 
assessment portion of the program at any time. While the instructional content was identical in each 
version of the program, informational screens were accessed differently in each of the three overview 
conditions. 

In the structured and unstructured overview conditions, all program screens included a 
navigational sidebar allowing direct access to any instructional node at any time. Navigation within an 
instructional node was achieved by clicking on next or back buttons. Location within the program was 
indicated within the sidebar through changes in the color of selected text, from black to blue. When an 
alternate topic was selected, the previously selected blue text returned to black. Sidebar topics were 
arranged hierarchically, from least to most complex, in the structured overview (see Figure 1) and 
alphabetically in the unstructured overview (see Figure. 2). The no overview condition had no sidebar, 
relying instead on text imbedded hotlinks to access informational nodes. Once activated, hotlinks changed 
color to indicate their use. Access to subsequent instructional screens was achieved either by clicking on 
additional hotlinks embedded in the instructional text or clicking on a next or back button. 

Figure 1. Structured overview 
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Figure 2. Unstructured overview 
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The demographic survey consisted of eight questions, four focused on general background 
information and four focused on computer experience. General questions solicited information on age, 
gender, class standing and major. Computer experience questions focused on computer use and 



25 



35 

BEST COPY AVAILABLE 



confidence, including years of computer use, hours of computer use per week, a self-rating of computer 
skills (beginner, intermediate, advanced), and a self-rating of confidence for success with computer related 
tasks (very confident, fairly confident, not confident). Responses to these four computer experience 
questions were combined into an overall computer experience rating (high, low) for each subject. 

Procedures 

During the first week of regularly scheduled class meetings, the vocabulary portion of the Nelson 
Denny Reading Test: Form H (Brown, Fishco and Hanna, 1993) was administered to all subjects. The 
vocabulary section has a time limit of 15 minutes for the completion of 80 items. Students recorded their 
answers on scantron sheets that were scored by computer. Test results produced an overall mean score and 
standard deviation of 60.53 (SD = 17.00). Descriptive data for the Nelson Denny standardization sample 
indicates a range of mean scores from 59.08 (SD = 1 3.97) to 63.20 (1 1 .98) for four year college juniors and 
seniors, the demographic group comprising over 70% of study participants. The reliability estimate for the 
vocabulary portion of the instrument is .89 (Brown et al., 1 993). Subjects were blocked by reading level 
(high, low) and randomly assigned to one of the three treatments. 

One week later, subjects were told that part of the course curriculum would be delivered via 
computer based instruction as a homework assignment. Subjects were then given a CD-ROM and floppy 
disk, along with directions on how to access the instructional program on the CD-ROM and record their 
results. Subjects were told the results recorded on the floppy disk would constitute part of their course 
grade, with 1 0 points awarded for completing the assignment, and 1 5 additional points awarded if their 
posttest score was 70% or higher. Subjects were given two days to complete the assignment, after which 
the CD-ROMs and floppy disks were returned to the class instructor. 

Criterion Measures 

This study utilized two criterion measures, a post-instruction achievement test, and an attitude 
survey. In addition, en-route data for instructional time was measured. 

Achievement was measured by a 22-item, fill-in-the-blank and multiple choice posttest conducted 
by computer immediately following the instruction. The first two questions were recall items worth five 
points each for listing the five outcomes of learning and the five types of intellectual skills. The next nine 
questions were application items requiring subjects to supply an appropriate action verb for the given 
instructional objective. An example of an application level fill-in-the-blank question is shown below: 

Click on the colored box, then type in an appropriate verb for the objective below: 

Students will pictures of sea creatures as mammals or nonmammals. 

The remaining 1 1 application questions were multiple choice items. An example of a multiple 
choice item is shown below: 

An English student correctly defines the term meter in poetry. 

Which of the following does that illustrate? 

a. higher-order rules 

b. concrete concepts 

c. defined concept 

d. verbal information 

Fill in the blank items were scored by the author, while multiple choice items were scored by 
computer. The maximum possible score was 30 points. The alpha reliability of the posttest was calculated 
as .86. Content validity was established based on the alignment of assessment items, instructional 
objectives and the instructional content of the CBI program. 

Data for attitude toward the program were collected through a 14-item, paper-based survey which 
included 1 1 Likert type questions and 2 constructed response items. The 1 1 Likert questions used a five 
point scale from Strongly Agree to Strongly Disagree. Internal reliability calculations for this portion of 
the survey produced an alpha of .85. Questions focused on such things as the difficulty of, value of, and 
feelings toward the program, as well as the student’s level of confidence about deciding where to go within 
the program. Example questions include, “I liked having the option to move around in the program” and 
“The navigation bar made finding information easy” for the structured and unstructured overview 
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conditions or “The hotlinks made finding information easy” for the no overview condition. The 
constructed response items asked what participants liked best and least about the program, 

En-route data were collected for instructional time then transferred to a data file on floppy disc. 

Data Analysis 

A separate 3X2 analysis of variance (ANOVA) was conducted on data for achievement and 
instructional time. Independent variables were overview mode (structured overview, unstructured 
overview and none) and computer experience (high and low), A 3 X 2 multivariate analysis of variance 
(MANOVA) was performed on data for attitude. The MANOVA for attitude included each of the survey 
items as a dependent measure. Follow-up univariate analyses (ANOVA) were conducted if significant 
multivariate results were found. Alpha was set at ,05 for all initial statistical tests. Alpha was reduced to 
.01 for follow-up univariate analyses of all significant multivariate attitude results. 

Results 

Achievement 

Posttest score means and standard deviations are shown in Table 1, Means for students with high 
computer experience were 18.62 (62%) for participants in the structured overview condition, 20,92 (70%) 
for participants in the unstructured overview condition, and 18,08 (60%) for participants in the no overview 
condition. Means for students with low computer experience were 17,50 (58%) for the structured overview 
, 17.73 (59%) for the unstructured overview, and 13.79 (46%) for the no overview condition. Means for 
the three overview modes were 1 8.08 (60%) for the structured overview , 19.1 5 (64%) for the unstructured 
overview, and 15.85 (53%) for the no overview condition. 
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Table 1 . Mean Posttest Scores and Standard Deviations by Overview Mode and Computer Experience 



Computer Experience 



Overview Mode 




Hi 


Low 


Total 


Structured 


M = 


18.62 


17.50 


18.08 


n = 25 


SD = 


6.37 


5.02 


5.67 


Unstructured 


M = 


20.92 


17.73 


19.15 


11 

c 


SD = 


6.36 


5.30 


5.90 


None 


M = 


18.08 


13.79 


15.85 


n = 27 


SD = 


6.87 


4.63 


6.11 


Total 


M = 


19.12 


16.32 


17.68 


n = 79 


SD = 


6.48 


5.21 


5.99 



Note . Maximum possible score = 30 

A 3 X 2 ANOVA (overview mode and computer experience) identified a significant main effect 
for computer experience, F (1 , 79 ) = 4.799, p < .05, ES = .47. Students with high computer experience (M 
= 19.12) performed significantly better than those with low computer experience (M = 16.32). ANOVA 
did not reveal a significant effect for overview mode, F (2, 79) = 2.352, p > .05, nor a significant 
interaction between overview mode and computer experience F (2, 79) = 0.499, p > .05. 

Attitude 

Means and standard deviations for each attitude survey item are shown in Table 2. The eleven 
Likert attitude items were scored on a five point scale from 1, most positive, to 5, least positive. A 3 X 2 
MANOVA indicated a significant main effect for overview mode, F (22, 120) = 2.21 , p < .05, but not for 
computer experience F (1 1 , 60) = 1 .00, p > .05. Furthermore, there was no interaction between overview 
mode and computer experience F (22, 120) = 1 .004, p > .05. 
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Table 2. Mean Attitude Item Scores by Overview Mode 



Overview Mode 



Item 


Structured 


Unstructured 


None 


I liked this program. 


2.76 


2.74 


3.01 


I would recommend this program to other students. 


2.64 


2.96 


3.00 


I preferred learning about verbs for objectives with a 
computer program rather than in a lecture format. 


2.76 


3.00 


3.37 


This program was easy. 


2.52 


2.96 


3.37 


I learned a lot about choosing verbs for objectives. 


2.76 


2.62 


3.15 


I tried hard to do well in this program. 


2.24 


2.19 


1.85 


The (navigation bar / hotlinks) helped me keep track 
of where I was in the program.* * 


1.88 


1.63 


2.74 


The (navigation bar / hotlinks) made finding 
information in the program easy.* 


1.80 


1.59 


2.78 


The (navigation bar / hotlinks) made moving through 
the program easy.* 


1.60 


1.41 


2.73 


I felt confident about deciding where to go in the 
program.* 


1.80 


1.89 


3.07 


I liked having the option to move around in the 


2.04 


1.56 


2.50 



program.* 



Note. Scale of 1 to 5 with 1 the most positive. 

* Significant differences, p < .01 

Follow up univariate ANOVAs revealed that overview mode had a significant effect on the 
following attitude items: The (navigation bar / hotlinks) helped me keep track of where I was in the 
program, F (2, 76) = 10.29, p < .01; The (navigation bar / hotlinks) made finding information easy, F (2, 

76) = 1 1 .09, p < .01 ; The (navigation bar / hot links) made moving through the program easy, F (2, 76) = 
16.89, p < .01 ; I felt confident about deciding where to go in the program, F (2, 76) = 9.09, p < .01; I liked 
having the option to move around in the program, F (2, 76) = 8.1 8, p < .01 . 

Post hoc Tukey analysis revealed that the means for participants in both the structured overview 
and unstructured overview condition were significantly higher than the means for participants in the no 
overview condition for all items listed above, with the exception of “I liked having the option to move 
around in the program.” For this item the unstructured overview mean was significantly higher than both 
the no overview the mean and the structured overview mean. 

Instructional Time 

Examination of instructional time data revealed two outliers with values two to four times that of 
any other data points. As the CBI was completed by participants individually on their own computers, these 
outliers were eliminated from subsequent time-in-program analysis on the assumption that they were likely 
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to represent instances where participants left the CBI running while otherwise engaged before returning to 
complete the program. 

The mean time-in-program for the three treatment groups was 16 minutes 30 seconds for 
participants receiving the structured overview, 14 minutes 44 seconds for participants receiving the 
unstructured overview, and 9 minutes 28 seconds for participants receiving the no overview condition. A 3 
X 2 ANOVA (overview mode and computer experience) identified a significant main effect for overview 
mode F (2, 77 ) = 3.46, p < .05. Post hoc Tukey analysis revealed that the mean for participants in the 
structured overview condition was significantly higher than the mean for participants in the no overview 
condition. ANOVA did not reveal a significant effect for computer experience F (1, 77) = .88, p > .05, nor 
a significant interaction between overview mode and computer experience F (2, 77) = .28, p > .05. 

Discussion 

The purpose of the study was to investigate the effects of overview mode and computer experience 
in a hypermediated learning environment. Subjects read a hypermedia unit that included either a structured 
overview, unstructured overview or no overview. The study examined the effects of overview mode and 
computer experience on achievement, attitude and instructional time. 

Achievement 

Results indicated that participants with high computer experience learned more from the hypertext 
program than those with low computer experience. Computer experience may influence achievement by 
reducing the cognitive load associated with learning from hypertext, thereby permitting the focusing of 
increased mental resources on the learning task. These results are supported by the findings of Swan et al. 
(1999) who observed that those with greater Internet experience moved more easily through information 
presented in hypertext form. The capacity to move through hypertext easily may reflect a reduced need for 
cognitive resources focused on navigation. Studies by Ayersman and Reed (1998) and Reed et al. (1996) 
further support a relationship between computer experience and cognitive load. Both reported that subjects 
with increased computer experience showed greater facility at recognizing hypertext structure. 

Recognizing a hypertext’s structure may reduce the mental resources needed to remain oriented in a 
hypertext environment. 

While computer experience was significantly related to achievement, overview mode did not 
influence achievement in the current study. A credible explanation for the lack of effect for overview mode 
may be the relatively simple structure of the instructional hypertext. The hypertext overviews contained 
five main headings, one of which included five subheadings. This lack of complexity may have allowed 
subjects to recall hypertext topics regardless of treatment. In a study of the effects of headings, topical 
overviews and topical summaries on recall Torch and Torch (1996) reported that such organizational 
signaling devices were unlikely to result in increased recall when the topic structure of text was relatively 
simple to encode and remember. 

Another possible explanation for the lack of effect for overview mode concerns the limited 
interaction between subjects and the overviews. Instructional times for all groups was relatively brief 
suggesting that time spent interacting with the overviews was minimal. Tovar and Coldevin (1992) found 
the positive effects of orienting activities on achievement was related to time spent in the program. It may 
be that the instructional time in the current study was too brief to support creation of a viable schema 
capable of enhancing recall. 

A final explanation for the lack of effect for overview mode relates to the finding that subjects 
with increased computer experience recognize hypertext structure more easily (Ayersman & Reed, 1998; 
Reed et al., 1996) The ability to identify underlying hypertext structure may eliminate the efficacy of an 
overview designed to make such structure clear. 



Results showed participants given structured or unstructured overviews had significantly higher 
mean attitude scores than those given no overview. Comments made by participants in the constructed 
response portion of the attitude survey suggest that overview mode may influence attitudes by reducing the 
level of frustration associated with navigating through the hypertext. Comments by those given an 
overview included such things as, “The navigation bar made things easier to follow”; “[The program] was 
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easy and efficient to move through at my own pace”; and, “It was easy to access information.” In contrast, 
those given no overview expressed frustration with navigation, recording comments such as, “I couldn’t 
figure out where to go”; “I thought if I went through the program I would get all the information.” These 
responses highlight the difficulty subjects with no overview encountered, supporting previous findings that 
subjects navigating by use of hot links often experience disorientation (Kozma, 1991; Lawless, 1996; 
McDonald and Stevenson, 1996; Rojewski, Gilbert & Hoy, 1994). 

Instructional Time 

Participants given a structured overview spent significantly more time on the program than those 
given no overview, results consistent with the findings of Tovar and Coldevin (1992). Comments made by 
several participants at the time of the study provide a clue as to why those with no overview spent less time 
on the program. They indicated they became so fhistrated in their attempts to locate and read all the 
instructional screens using hotlinks that they eventually moved on to the assessment despite feeling that 
they failed to read all program content. 



Implications 

The use of instructional hypertexts continues to expand rapidly as a variety of institutions offer 
increasing numbers of web-based educational opportunities. The present study has implications for 
practitioners developing instructional hypertext. Results suggest that establishment of prerequisite 
technical ability or computer experience for students enrolling in hypertext based instruction may 
contribute to student success. Additionally, inclusion of an overview may contribute to more favorable 
student attitudes. 

Future Research 

As the rush to provide greater online educational opportunities through the use of hypermediated 
instruction continues, it is important that educational hypertext designers identify those factors contributing 
to positive outcomes. Future research should continue to investigate the role computer experience plays in 
learning from hypertext through identification of those aspects of computer experience responsible for 
supporting recall such as years of computer use, weekly hours of computer use, or computer ownership. 
Additionally, the effect of headings on recall warrants further attention given that headings are a common 
feature on many educational web-sites. The current study used a relatively simple hypertext structure and 
assessment focused predominantly on questions at an application level. Future research incorporating more 
complex hypertext structures and assessment focused on recall may further illuminate the relationship 
between headings and recall, thereby supporting the design of more effective hypertext based learning 
experiences. 
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The Use of CSCA to Support Argumentation Skills in 

Legal Education 



"/ choose the word ''argument'' thoughtfully, for scientific demonstrations, even mathematical 
proofs, are fundamentally acts of persuasion. Scientific statements can never be certain: they can 
only be more or less credible". - Joseph Weizenbaum in Computer Power and Human Reason. 
1976 

Introduction 

According to the MacCrate Report (Blasi, 1995, p. 314), legal problem-solving is the 
“fiindamental lawyering skill,” listed first among those skills the American Bar Association task force 
deems important. The ability to construct and organize an argument in support of some position is the 
central intellective ability in solving [legal] problems (Cerbin, 1988). As a result, law schools are 
beginning to introduce courses on general ‘lawyering skills’, which include the responsibility of teaching 
legal argument (Blasi, 1995). But how should we teach legal argument? 

Teaching Legal Argumentation 

Practicing student-to-student interaction and argumentative dialogue have been found to be 
positively linked with argumentation and critical thinking skills (Hart, 1990; Marttunen, 1992; Smith, 

1977). Further, Buckingham Shum et.al. (1997) argue that exposing an argument's structure facilitates its 
subsequent communication since important relationships can be more easily perceived and analyzed by 
others (Buckingham Shum, et. al 1997). In this study, a Computer-Supported Collaborative Argumentation 
(CSCA) is used as a tool to help teach legal argumentation. CSCA supports argumentation through student- 
to-students interaction and by exposing an argument’s structure as a means to focus argumentative 
dialogue. 

Argumentation is a process of making assertions (claims) and providing support and justification 
for these claims using data, facts, and evidence (Toulmin, 1958). The goal of legal argumentation is to 
persuade or convince others that one's reasoning is more valid or appropriate. Toulmin's model of 
argument provides the language symbols that support the argumentation process (see Figure 1.1). 
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Fi^rel Toulmin's \4ndel of A raiment 

Toulmin’s model is procedural, not static or spatial (Toulmin, Ricke, & Janik 1984). Based on 
legal reasoning, the layout of his argument model focuses on the movement of accepted data through a 
JarraM, to a claim. Toulmin recognizes three secondary elements that may be present (and sometimes 
implicit) in an argument; backing, qualifier, and rebuttal. Backing is the authority for a wamnt, Fovdes 
credibility for the warrant and may be introduced when the audience is unwilling to accept the warrant at 
face value. A qualifier indicates the degree of force or certainty that a claim possesses; it converts the 
terms of the argument from absolute to probable. Finally, rebuttal represents certain conditions or 
exceptions under which the claim will fail; it anticipates objections that might be advanced against the 

argument to refute the claim (Toulmin, 1958). , j ^ 

In the following example, an argument with regard to a specific legal case is represented using 

Toulmin's model of argument (see Figure 2). 



backing 

Because of 
mandatory 
precedent 



DATA 

P was terminated without 
reason. The employment 
handbook states that 
termination will occur for 
insutwrdination or poor 
performance. P was not 
insubordinate and had recently 
received a good performarKe 
review 



I 

provides authority to 

T 



QUALIFIER 

Presumably 



‘.'.eves ihrougn a ^ 



WARRANT 
Failure to terminate an 
employee for a cause 
specified in the employment 
handbook results in the 
employer's liability for 
damages for breach of 
employment agreement 



Doscriocs 



; suDport a ■ 



CLAIM 
Employer is liable 
for breach of 
employment 
agreement 



Presents an exception to 



. REBUTTAL 
Unless P was an 
■* employee at will 



Figure 2: Example ofle^al argument u.sine To ulmin's model 

A woman has been dismissed from her position and met with a lawyer to discuss a 

lawsuit. The lawyer suggests a cause of action for breach of employment agreement t at in st ercic 

sought with the facts of the woman's ease. The lawyer will support the cause of action with mandatoi7 
precedent, and will recognize that the employer may assert a defense based on the emp oymen a w 

doctrine that may deny recovery (adapted from Saunders, 1994). 

As such, Toulmin’s model of argument becomes a mechanism for structunng argumentation 
between law students. It aims to clarify reasoning by encouraging parties to make explicit important 
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assumptions, distinctions, and relationships as they construct and rationalize ideas (Buckingham Shum et 
al., 1997). 

CSCA to support Legal Argumentation 

In this study, Questmap™ was used to support legal argumentation by equipping the students with 
the language needed to construct and analyze arguments (see Figure 3). 




Fi 2 ure 3 . Sample CSCA structure including multiple components ofToulmin *s Model of Argument, 
recording using Questmap™. In this example, there are two claims (labeled, 'proposals ’ in accordance 
with legal terminology), several warrants, with backing (labeled, 'arguments ' and 'counterarguments ), 
and grounds or data (labeled, 'evidence ). This is a rudimentary example; legal arguments can become 
extremely complex and elaborate. 

Though Questmap™ (see Conklin, 1993) incorporates slightly different terminology (i,e,. 

Question, Idea, Pro, and Con), the law students were able to rename the icons to represent argumentation in 
a legal context. Figure 3 shows the language used by the law students. In this structure, a problem is a 
statement of an unknown or something that is unsettled, such as a controversy, an issue, etc. The problem is 
introduced by the professor in the form of a question (e.g., should this evidence be allowed in court?). A 
proposal is a recommendation for action in response to the problem (e.g., it should be admitted). Finally, an 
argument is comprised of evidence that supports a proposal (because of mandatory precedent). Thus, to 
elaborate on a problem, students submit proposals with supporting arguments. Additionally, students can 
create links (e.g., supports, contradicts, competes with, etc.) that indicate relationships between arguments. 
Students can then respond further with counter-arguments and rebuttals. Finally, the professor will 
announce a decision that serves as his summary judgment in the instant case after considering all relevant 
arguments. 

Method 

Throughout the course, students were assigned five problems to solve (one every other week). The 
control group completed them in study groups, using traditional legal resources, such as Westlaw™, legal 
texts, and library resources. After discussing the problem, the students would prepare one summary 
statement in narrative form that represented the consensus of the group. In addition to the traditional legal 
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resources, the treatment group was also allowed access to Questmap^*^ to complete the assignments in 
study groups. The students within the treatment group each logged into Questmap™ on their own computer 
and worked in a shared ‘virtual’ space with their study group (such that each person within a study group 
could view and modify contents submitted by other group members) and were seated beside one another to 
facilitate communication. 

Students adopted different methods for constructing arguments, but a general pattern emerged. 
During a typical Questmap'^*^ session, a study group proceeded as follows: (1) read the problem, (2) 
discussed the problem orally, (3) discussed what rules of evidence may apply, (4) volunteered to ‘map out’ 
certain rules and argue how they would apply, (5) began mapping arguments (arguments appeared in each 
students shared workspace instantly), (6) read other group members’ arguments, (7) responded to other 
arguments with rebuttals, (8) orally discussed issues such as, “Good idea. . “I didn’t think that would 
apply here, because. . .”, and “Wow, that’s exactly what this is about [in response to a posting by a fellow 
group member]...”, (9) organized the final map on the screen for printing, (10) printed the map view, (1 1) 
printed the outline view of the map. 

Throughout the semester, students were asked to submit responses to each problem including their 
arguments, expected counterarguments, and rebuttals. The control group handed those responses in on 
paper while the treatment group recorded their responses graphically using Questmap™. The graphs were 
then automatically converted to outline form and printed out for submission (see Figure 4). 
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Sample Outline Created 

1 ; Predisposition to kill 

vw^wvwwww * 

You are an assistant district attorney involved in the prosecution of the "crime of the year." Noted 
psychiatrist, Dr. Murray Slaughter, is accused of murdering his wife, Marie, by poisoning her. One of 
your private investigators discovered a tape of a radio appearance that Dr. Slaughter made last year on a 
local radio call-in show. The theme of the show that is heard on the tape is people whose rage is so 
strong it could drive themto kill someone they claim to love. 

^The trial starts onTuesdey. Your supervisor wants to know if the prosecution can call the 
investigator to repeat what he heard on the tape. Discuss all issues. 

1.1 Idea (Responds To Question 1 ): admissOile 
(peter) 

| .|.l (Supports Idea 1.1 ): 401-402 - Relevant 

The comment, "If I were ever to kill my wife, I would poison her...," seems relevant 
to show that the Defendant had apian or a particular method of killing his wife in 
mind. [401-402] The statement makes more probable than not that the doctor had in 
his mind the killing of his wife. 

1 .1 .1 .1 Pro rObiects T o Pro 1 .1 .1 '1 : 403 - Preindicial 



Figure 4: Sample Ouestmav'^^ Map and corresponding Outline 

Finally, the professor would generate the optimal solution and distribute it to the control group in 
written form and the treatment group in Questmap'^'^ form. 

After completing these five problems throughout the semester, a final exam was administered to 
assess the effectiveness of argumentation skills. A comparison of final exam scores (control vs. treatment 
group) was conducted to assess the effectiveness of the CSCA tool in developing argumentation skills in 
second-year law students over the course of one semester. 
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Results 



The results of this study are discussed in terms of efficiency and effectiveness of CSCA use in a 
legal education. 

Efficiency 

In a study of CSCA use by instructional designers, Buckingham Shum (1994) describes a situation 
in which the argumentative mode of reasoning proved incompatible with the construction task at hand. A 
designer encountered “severe difficulties in breaking another problem into discrete Questions, Options and 
Criteria” (the names of the primary argument components used in this instance). 

Alternatively, in this study, law students reported an increase in the efficiency in which they were 
able to complete assignments (Carr, 1 999). Further, the complexity of the maps (as determined by the 
number and types of nodes present) did not significantly increase through continued use of Questmap™, 
suggesting that students did not encounter such ‘cognitive overhead’ as reported by Buckingham Shum 
(1994) in a study involving instructional designers. Not only were the students able to complete individual 
assignments more efficiently, they were able to automatically generate outlines from which to study for the 
final exam. 

Effectiveness 

An assumption that the use of Questmap™ would support the development of argumentation skills 
in second-year, second-semester law students proved false. The results of this study indicated that the 
treatment group did not have a significantly different score (f=0.05, 6{^69.210,/?>0.05) on the practice final 
than the control group. 

Since these students were all second-year law students and did not have any real difficulty in 
learning how to use Questmap™, it is suspected that the students were already very good at legal argument. 



Conclusion 

The use of Questmap™ supported the efficient creation of arguments among groups of second- 
year law students throughout a legal education course. Further, Questmap™ maps could easily be 
converted into an outline format and printed to help students study for final exams. 

Nearly all students reported that Questmap™ sessions helped them maintain a focus on the task at 
hand, providing for structured argumentation sessions. Further, they were able to benefit from the 
comparison of their argumentation maps with those created by the professor. In an exit interview, one 
student commented: 

For me, it got me in here every week to review. I thought that was great... Plus, 
[Questmap] gives focus for our study group. Instead of interrupting us with 
conversations about whafs going on in our lives, the computer focuses us and keeps us 
on task... Also, it is a way to compare our analysis with [the professor's] which is 
basically, a correct analysis. 

Further, students attested that [using Questmap™] was much faster than meeting in groups for 
several reasons: (1) it focused group discussion, (2) it provided for adversarial role-playing, and (3) it 
provided a means for efficient storage and retrieval of arguments throughout the semester. Shum et. al 
(1997) reported similar results with designers using CSCA (see Table 1). 



4 8 



39 



1 



Table 1: Benefits of CSCA (adapted from Buckingham Shum et. al. 1997) 





Collaborative 

Argumentation 


Store information 


serving as a long term, reusable 
project memory 


Express ideas 


supporting the emergence and 
debate of new ideas 


Mediate interaction 


structuring and focusing 
discussion 



Buckingham Shum et. al., (1997) classified the overall benefits of CSCA to designers into three 
important functions: (1) to store information (Questmap’^’^ stored all entries in a central database. Users 
could browse old argument structures and search them for keywords), (2) to express ideas (Questmap’*'*^ 
supported the ability to express ideas using common legal terminology (e.g, objects to, supports, etc.)), and 
(3) to mediate interaction (students reported less ‘small talk’ and more intense focus on argumentation 
throughout weekly Questmap’^^ sessions). 



Future Research 

No significant differences in final exam scores were found between the treatment and control 
groups indicating that Questmap”*”^ was not effective in increasing argumentation skills in second-year, 
second-semester law students. Further research should investigate the level of expertise at argumentation 
in which CSCA can be effective, yet still reasonably efficient. Since the results yielded a marginally 
significant result (/=0.05, c^69.210,p>0.05) with students highly skilled in argument, it is suspected that 
the use of Questmap'*'^ by students with less expertise in argumentation would provide significant results. 
Indeed, expert vs. novice studies in the field of economics have shown argumentation skills to be positively 
linked with level of education (Voss, et. al., 1986). 
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Character versus Narration: Empirical Investigations 
OF Drama’s Role in Multimedia Instructional Software 

Ward Mitchell Cates, Mary Jean Bishop, & Woei Hung 
Lehigh University 



Abstract 

As part of an ongoing research program, the researchers investigated the use of single-voiced 
narration and multi-voiced characterizations/monologues in two studies of an instructional 
program on information processing that employed metaphorical design. The results suggest that 
single-voiced narration is at least as good as multi-voiced characterizations/monologue, and may 
in fact be superior The iterative process by which the researchers refined design of the two 
versions, the bases for their conclusion, practical implications, and future areas of research are 
discussed. 

Designers and developers of instructional software are constantly seeking new ways to engage 
learners because users of engaging instructional products appear to acquire and retain more of the content 
provided (see Csikszentmihalyi, 1979, 1990; Hannafm, 1989; Jacques, Preece, 8l Carey, 1995; Jones, 1997, 

1 999; Perrone, 1 994). One design approach aimed at gamering such student interest is use of the 
metaphorical interface (Cates, 1993, 1994, 1996). It is unclear, however, which aspects of metaphorical 
design hold most promise for capturing student interest and increasing engagement. 

Because metaphorical design often employs characters, the researchers have been investigating 
ways to enhance learner engagement (and content recall and retention) by looking at metaphorical design 
through the prism of characterization. The researchers based the two studies described below on Oren, 
Salomon, Kreitman, and Don’s (1990) findings that characterization enhanced learners’ engagement with 
the information presented in an educational hypermedia product. In fact, the authors found that even when 
no attempt was made to develop screen entities as characters, learners still attributed personalities to them. 
Don (1990) suggested that because screen characters help to “collapse distinctions” between the interface 
and the content, they may supply learners with a more directly-engaging experience with the material under 
study (p. 390). Laurel (1990) hypothesized that screen characters capitalize on users’ ability to infer 
thoughts and actions based on external traits, creating a sort of “cognitive shorthand” that helps users 
understand the interface (p. 358). 

Characterization is used in this context to refer to the development of each individual shown on 
the screen in a way that distinguishes that individual from all others and imparts a distinctive and 
identifiable personality {Webster's New Collegiate Dictionary, 1976, p.l87). In writing, characters are 
often developed through narrative descriptions of their actions, their thoughts, and what others think about 
them (Dietrich &. Sundell, 1974). While narrative forms have the flexibility to supply a number of 
perspectives from an “omniscient” narrator, Beckerman (1990) argued that telling about a character is often 
more cumbersome and less intense than showing a character. Plays and films have the expressive 
advantage of being able to develop characters through actions and speech, musical themes, and visual and 
auditory cues from the setting (Booth, 1 983). Recent advances in digitized audio and video technologies 
and their widespread availability mean that screen characters in instructional software also may now be 
developed in the “first person,” auditorially as well as visually. 

First-person sounds available to develop characters in instructional software consist of sound 
effects, music, and speech. Sound effects can be either concrete or symbolic. Concrete effects illustrate 
realistically the sound with which the learner already associated something represented on the screen. 
Symbolic sound effects establish a new convention — or use a pre-existing convention — for associating a 
non-concrete sound with something represented on the screen. For example, if one wished to use a sound 
effect to attribute great speed to a screen character, one might use the concrete sound of the rapid footfalls 
of a runner or a symbolic sound like a loud “whoosh.” Thomas and Johnston (1984) suggested that 
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accompanying music might be used to influence the affective properties learners associate with a character. 
According to Giles and Street (1994), however, of all the sound types, speech is the most likely to 
communicate a screen character’s cultural background, demeanor, thoughts, personality, and traits. Carr 
(1986) proposed that, in addition to sound’s established vocabulary and syntax, the rhythm and intonation 
patterns of speech provide a number of syntactic and referential cues not easily conveyed through other 
sensory phenomena. Consequently, as Griffiths (1984) and Zettl (1990) noted, speech is among the most 
popular ways to convey information about character. First-person character speech can be either dialogue 
(when two or more characters speak to one another) or monologue (when a single character does all of the 
speaking) (Bain, 1973; Hunter, 1973), 

But while it seems that characterization through first-person sound may hold promise for the 
design of more engaging lessons, the authors of instructional design guidelines seem to have focused 
almost exclusively on third-person narration of screen text (see, for example, Alessi & Trollip, 1991; 
Fleming & Levie, 1993). Similarly, studies investigating sound’s use in computerized instruction have 
analyzed narration of content and narration’s effects on achievement, time on task, and attitudes toward the 
software (see, for example, Barron & Atkins, 1994; Barron & Kysilka, 1993; Mann, 1994, 1995, 1997; 
Shih& Alessi, 1996). 

For this reason, the researchers decided to focus on the extent to which auditory narration and 
characterization appear to affect learners’ recall and retention of content differentially in a metaphorical 
instructional program. With so little existing theory and prior research to guide the researchers, they further 
decided to begin exploring auditory characterization in computerized instruction through an iterative 
process of software development and modification, data collection and analysis, theoretical refinement, and 
product revisions, as recommended by Savenye and Robinson (1996), In the sections that follow, the 
researchers describe the first two of these studies and consider their implications for researchers and 
developers. 

Study One: Characterization versus Narration 

This initial study was designed to test whether a characterized or narrative auditory form would be 
more effective for learning from a metaphorical instructional program. The characterized version of the 
program utilized individually voiced monologues, while the narrated version used a single-voiced narrator. 
The researchers hypothesized that the use of separately voiced monologues might enrich learners’ 
perceptions of individuality and depth for each of the characters in the program, thus engaging the learners 
more deeply in the processing of the lesson’s content. In addition, the researchers conjectured that, if such 
deeper processing occurred, learners would retain more information about those characters and the 
interface in general. 

Methods 

Sample 

The sample for this study consisted of thirty-four (5 males and 29 females) undergraduate and 
graduate students enrolled in a Child Development graduate class at a private mid-size university in 
northeastern United States. Participants ranged in age from 19 to 39, with a mean age of 24.4. The 
researchers randomly assigned participants to two different treatment groups: one with individually-voiced 
characterizations and sound effects (experimental group) and one with single-voiced narration and no 
sound effects (control group). Of the original 36 subjects in the study, 2 were dropped, one from each 
treatment group. Although these two participants completed all parts of the treatment and all instruments, 
their data were excluded from subsequent analysis because both proved to be outliers. One, a 24-year-old 
female Caucasian graduate student from the characterized treatment group, scored 2.4 standard deviations 
below her group mean on the posttest. The other, a 30-year-old Asian female graduate student from the 
narration treatment took 2.85 standard deviations longer to complete the treatment. This appeared to be a 
language problem, a conclusion supported by observations of study monitors, statements she made about 
difficulties she was having understanding the language in the program, and questions she asked study 
monitors. No other participants were lost. 
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Instrumentation 

The researchers designed five measurement instruments, including a pretest survey, a posttest, a 
posttreatment questionnaire, a retention test, and delayed questionnaire. The pretest survey gathered the 
participants’ demographic information, prior knowledge about the information-processing model, ratings of 
instructional software, and attitudes toward using instructional software in teaching. The posttest measured 
participants’ recall of program content and interface properties (names of characters, colors employed, and 
the like) immediately following completion of the treatment. The posttreatment questionnaire asked 
participants to rate the value of instructional software and express their attitudes toward using instructional 
software in teaching. The retention test assessed participants’ retention of the content and properties of the 
interface. The delayed questionnaire asked them once again to rate the value of instructional software and 
express their attitudes toward using instructional software in teaching. 

Items on the retention test were reordered to control for item presentation-order effects. To control 
for testing sensitization, posttest items that called for the learner to assign events to one type of memory, 
episodic or semantic, were replaced on the retention test by alternate, but equivalent, forms of the items. 
Similarly, open-ended items that called for reflection by learners on the posttest were replaced by alternate 
equivalent items on the retention test. Since no discemable pattern existed in learner responses to such 
open-ended items emerged in analysis, they are not discussed in this present report. 

The reliability of measurement instruments was tested. Test-retest reliability (posttest to retention 
test) was .657 (j?<.001). Coefficient alphas for rating-of-instructional-software items were .869 for the 
pretreatment survey, .684 for the posttreatment questionnaire, and .893 for the delayed questionnaire (all 
p<.05). Reliability calculations for items related to attitudes toward using instructional software in teaching 
produced coefficient alphas of .599 for the pretreatment questionnaire, .63 1 for the posttreatment 
questionnaire, and .780 for the delayed questionnaire (all p<.05). 

Treatments 

After examining a variety of possible metaphors to serve as the basis for a lesson on the 
information-processing model, the researchers finally agreed to use a package-shipping center as the 
underlying metaphor. In a metaphorical graphical user interface, the underlying (or primary) metaphor is 
the metaphor upon with the program is based and to which all auxiliary metaphors relate. Auxiliary (or 
secondary metaphors) are subsequently introduced metaphors designed to complement and enrich the 
underlying metaphor (Cates, 1 994). The package-shipping center metaphor appeared to meet the 
instructional demands of the program to be designed and mapped well to the content to be delivered. The 
researchers used a POPIT analysis (Cates, 1994) to generate a sufficient number of auxiliary metaphors to 
contribute to a rich metaphorical graphical user interface. 

The researchers created two versions of an instructional program using the package-shipping 
center as its underlying metaphor. Both versions addressed information-processing theory’s explanations of 
how the brain processes, stores, and recalls stimuli. Both versions shared identical screen images, 
navigational controls, animations, and screen text. Because the program was to be used for research, each 
version allowed learners to visit a screen only once; there was no backward navigation. Each version of the 
program divided the content into the same four major blocks: (1) an introduction to the “staff’ of the 
package-shipping center; (2) an animation in which the learner watched as packages came into the package- 
shipping center, with accompanying auditory explanations of what was happening in terms of package- 
processing; (3) a series of explanatory (elaboration) screens related to the individual stages of information 
processing, accessed through user selection from a graphical menu; and (4) an exact repetition of the 
animation in which packages came into the package-shipping center, this time with accompanying auditory 
explanation of what was happening in terms of information processing. The program’s animation sequence 
was designed to act as a mental model for information processing (Gardner, 1 985). 

Where the two versions differed was in the way in which they used sound. In the narration 
version, the introduction to the staff was accomplished through a single-voice narrator who identified each 
character and described that person’s responsibility in the package-shipping center when the learner clicked 
on the cartoon image of that character. In the characterization version, when the learner clicked on each 
staff member’s cartoon image, that staff member introduced himself or herself; each of the five characters 
had a different voice. In the subsequent animation sequences (blocks 2 and 4), in the narration version the 
same single-voice narrator described what was occurring in the animation, while in the characterization 
version each of the five characters explained what was occurring in his or her portion of the animation. The 
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issue of where to use sound effects in the animation was a thorny one. After much discussion, the 
researchers decided that sound effects might enhance the sense of characterization and that including sound 
effects in the narration version might confound treatment differences. For this reason, only the 
characterization version used sound effects. Music was excluded from both versions of the program to 
eliminate any contributory interactions. 

Procedures 

The two treatment sessions took place in a university computer lab equipped with 1 8 PowerMac 
8600 computers, each with 32MB RAM, a 17 inch color monitor, and a set of earphones. Each computer 
was labeled with a participant’s name. Name labels were changed between treatments. Both versions of the 
information-processing instructional program were preinstalled on all computers in the lab. Before each 
treatment group began its treatment session, one of the researchers assured that each computer was ready to 
begin the appropriate treatment. 

In a pretreatment session, the researchers explained the purpose and procedures of the study and 
distributed informed consent forms to the participants. After signing consent forms, participants completed 
a pretreatment survey. A coin flip determined the narration group would receive treatment first and the 
characterization group second. Once all participants completed the pretreatment survey, the narration group 
accompanied one of the researchers to the computer lab, while the characterization group stayed in the 
classroom and worked with the course’s instructor on subject matter unrelated to the study. 

Once in the lab, participants went to their labeled computers. When all the participants were ready, 
the researcher began timing participants as they worked with the program. As participants finished, the 
researcher gave them posttests and questionnaires to complete. The researcher recorded when participants 
completed the program and when they handed in the posttest and posttreatment questionnaire, and the 
researcher and a study monitor looked for any problems during treatment. After all narration group 
participants completed treatment and instruments, they returned to the classroom and the characterization 
group came to the lab to complete treatment using the same procedure. In order to avoid contaminating the 
second treatment, participants in the narration group were cautioned not to talk to members of the other 
treatment group as the two groups passed in the hall (Cook 8l Campbell, 1979). Two weeks later, 
participants completed a retention test and delayed questionnaire at the beginning of their regular class. 

Results 

Content 

An independent-samples /-test revealed no significant difference between groups in participants’ 
recall of content immediately following treatment [t(32) = .09,/? = .93; characterization mean = 9.00, 
narration mean = 9.06]. There was also no significant difference in content retention two weeks later [t(32) 
= .56, p = .58; characterization mean = 8.12, narration mean = 8.47]. 

Both treatment groups exhibited reduced content retention posttest to retention test 
[characterization mean difference: -.88; narration mean difference = -.59]. Paired-samples t-tests showed 
this forgetting was significant for the characterization group [t(16) = 2.43,/? = .03], but not for the narration 
group [t(16)= 1.37,/7 = .19]. 

Interface 

T-tests for independent samples compared participants’ recall of interface properties. No 
significant differences between characterization and narration groups on either the posttest [t(32) = 0, 
p>.99\ characterization and narration mean = 6.1 8] or the retention test [t(32) = .1 1,/? = .91 ; 
characterization mean = 5.53, narration mean = 5.47] were found. 

Once again, the two treatment groups differed in forgetting from posttest to retention test. In 
contrast to the pattern revealed in content recall, recall of interface properties exhibited the opposite pattern. 
There was a significant decrease in recall of interface properties for the narration group [t(16) = 2.51,/? = 
.02; mean difference = .71] but a non-significant decrease for the characterization group [t(16) = 1 .40,/? = 
.18, mean difference = .65]. 

Participants’ interface character-recognition scores and content scores did not correlate 
significantly (+.28,/? = .26 for the characterization group and +.17,/? = .50 for the narration group). 
Retention test correlations were +.24 {p = .35) for characterization and -.06 {p = .80) for narration. 
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Ratings and A ttitudes 

The narration group showed significantly higher mean ratings of instructional software from 
pretreatment to delayed questionnaire [1 1,06->1 1.71 ->12,47; t(16)=3.59,/?=,002], while the 
characterization group’s mean ratings remained stable [1 1 ,00-> 10,76-> 10,94; t(16)=,14,/?=,89]. Despite 
this within-groups difference, there was no significant difference in ratings of instructional software 
between groups before treatment [t(32)=,073,/?=,94], immediately following treatment [t(32)=l,45,/?=,16], 
or one week later [t(32)=1.91,/7=07]. 

Neither group showed significant changes in mean expressed attitude toward the use of 
instructional software in teaching from pretreatment to delayed questionnaire [narration: 
24.53->25.76->25.59, t(16)=1.99,/7=.06; characterization: 25,29^25,12^25,12, t(16)= .25,/?= ,80], 
Similarly, there was no significant difference between treatments in attitudes toward using instructional 
software in teaching before treatment [t(32)=.67,/?=.5 1], immediately following treatment [t(32)=,61, 
/?=,55], or one week following treatment [t(32)= .34, p = .74], 

Time-to-completion 

Analyses of time-to-completion revealed that participants in the two treatment groups took 
different lengths of time to complete their treatments [narration mean =18,1 minutes; characterization 
mean =16.1 minutes]. Slight differences between treatments in the length of digitized speech files used in 
the two animation sequences do not account for this two-minute differential (narrated block 2 was 12,59 
seconds shorter than characterized version, while narrated block 4 was 17,68 seconds longer than 
characterized version, for a total difference of only 5.09 seconds). The narration group spent significantly 
more time completing their self-instructional program than the characterization group [t(32)= 2,19, /? = 
.04], Further, the narration group spent significantly more time completing the posttest than did the 
characterization group [t(32) = 4.75,/? < .001; narration mean = 13.3 minutes; characterization mean = 8,0 
minutes], A Pearson Product Moment Correlation for all participants revealed a significant correlation 
between treatment time-to-completion and retention test content scores [r = +.383, p = .025] but not with 
other test scores (posttest content score, posttest interface score, retention test interface score). Similarly, 
treatment time-to-completion and how long it took participants to complete the posttest were significantly 
correlated [r= +.52,/? = .002], 

Interpretation 

Characterized and narrative auditory forms appear equivalently effective in producing immediate 
recall of content delivered by a metaphorical instructional program. The researchers’ hypothesis that 
individually voiced monologues would enhance content retention does not appear supported by the 
findings. Even recall and retention of interface properties do not appear to differ significantly between the 
auditory approaches. While the researchers conjectured that characterization might enhance subsequent 
recall of interface properties, which in turn might enhance content recall, there was no significant 
correlation between interface property scores and retention of content. In short, considering all the key 
variables the researchers sought to explore, it looks as if the two auditory approaches were equivalent. 

Since the researchers worked hard to produce two versions based on comparable and strong instructional 
designs, this may once again confirm Clark’s (1983, 1994) contention that good design matters more than 
medium or technique. 

Closer examination of the findings raises some interesting questions, however. For example, why 
did the characterization group’s content scores drop significantly between the posttest and retention test, 
while the narration group’s content scores did not? Why was there a significant decrease in recall of 
interface properties between posttest and retention test for the narration group but not the characterization 
group? Why did the narration group’s ratings of instructional software become significantly more positive 
over time, while the characterization group’s remained relatively stable? Given that the two treatments 
were largely equivalent in length, with only a 5 second difference attributable to narration differences in 
animations in blocks 2 and 4, why did the narration group spend significantly more time completing the 
lesson and the posttest than the characterization group? These differences seem to suggest as-yet- 
undisclosed differences in cognitive processing and learner reception of these auditory approaches. One set 
of interpretations is positive: Some property of the narration treatment seems to have enhanced retention 
and engendered a more favorable attitude. This same property, or some other, may have led either to more 
active learning behavior or to deeper processing of the content presented. The halo effect (Ormrod, 1990, 
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p.l93) then caused the narration group’s more favorable attitude to spread to their ratings of instructional 
software in general. A second set of interpretations is negative: Some property of the characterization 
treatment seems to have impeded retention and engendered a less favorable attitude. This same property, or 
another, may have led to either less active learning behavior, or to shallower processing of the content 
presented. The halo effect then caused the characterization group’s less favorable attitude to spread to their 
ratings of instructional software in general. Given the bipolar nature of these possible interpretations, the 
researchers decided to reexamine their assumptions about dramatic approaches to metaphorical design. 

Characterization or Drama? 

In this study, we set out to examine the differential effects of auditory characterization and 
narration on learning in order to determine how better to design the characters that are so often a part of 
metaphorical interfaces. But our distinction between characterization and narration may have assumed too 
tight a view. It may be, instead, that the reason interface metaphors often employ screen characters is that 
metaphorical designs lend themselves to dramatic representation. 

Like metaphor, drama works by comparing one thing to another without directly stating the 
comparison. One may think of drama in terms of the divisions in a traditional play. That is, a play is made 
up of acts; acts are made up of scenes; and scenes involve settings, characters, and action. Projecting this 
subdivision on a metaphorical interface, one may think of the underlying metaphor as the play, the 
individual complementary or auxiliary metaphors as the acts, and individual screens through which content 
is delivered and with which students interact as the scenes. Each scene would be made up of (a) the 
situation or background of the individual screen, (b) the actions and interactions that the learner would 
participate in or observe on that screen, and (c) any characters/actors illustrated on that screen. 
Characterization, therefore, is just one of many components of drama. 

Components of Drama 

Applying classical Aristotelian dramatic theory. Laurel (1993) outlined six elements of structure in 
human-computer activity: enactment, pattern, language, thought, character, and action. Taken together, 
these elements form a qualitative hierarchy. Moving from bottom to top, each element constitutes an 
evolution of its predecessor and suggests the next step in the process of refining the software “production.” 
Each level, as described by Laurel and applied to instructional design, is discussed briefly below. 

Level 1 : Enactment — the sum of sensory stimuli or “raw material” of a computerized lesson. At 
the enactment level, visual, auditory, and tactile phenomena are carefully chosen to enhance the learner’s 
perception of the lesson as a whole. 

Level 2: Pattern — the pleasurable ordering of the enactment’s stimuli into arrangements that are 
harmonious and complementary. Humans are, by nature, pattern-seekers (Ormrod, 1990, pp. 129-132). 
While unexpected perception of pattern can be pleasurable in and of itself, unexpected lack of pattern can 
be disappointing and disruptive. Balanced screen layouts, internal consistency, use of complementary 
colors, harmonic and appropriate use of sound, and inclusion of stimuli at comparable levels of “resolution” 
may enhance learners’ perceptions of pattern. Internal consistency refers to the consistent use of stimuli 
with comparable intents and for comparable purposes throughout the program. Comparable levels of 
resolution means that stimuli are of the same type and general quality, are appropriate to other stimuli with 
which they are used, and are used in consistent ways throughout the program. 

Level 3: Language — the semiotic (symbol-making and communicating) arrangement of the 
enactment’s stimuli. In instructional software, language can be any stimulus — verbal or nonverbal — that 
supplies the material for thought. That is, any graphical icon, nonverbal sound effect, or animation 
sequence that communicates and expands upon concepts used at the higher structural levels can be part the 
language of a computerized lesson. The “vocabulary” and “syntax” of such nonverbal languages often must 
be established before their symbols can express thought, however. For example, once learners have learned 
that a “piece of paper” icon represents a file and that a “folder” icon represents a directory, an animated 
piece of paper flying from one folder to another can communicate quite effectively the concept of moving a 
file from one directory to another. 

Level 4: Thought — the sum of processes (cognitive, emotional, rational, logical. Boolean, and the 
like) that lead to choices and actions. While thought may be explicitly expressed, in instructional software 
thought generally is inferred from choices and actions. Thus, while a computer may not actually think, a 
learner can infer thought from the stimuli presented by the program. If a program is able to arrange its 
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stimuli in such a way as to enhance the learner’s perception of “thoughtfulness,” the organic unity of the 
production may be enhanced. 

Level 5: Character — the bundle of predispositions and traits that describe agents of both human 
and computer origin in a human-computer activity. Like thoughts, the internal traits that dictate how an 
agent can or will act are inferred from the way a program represents those predispositions externally. Well- 
developed computer-based agents exhibit external traits that are consonant with the agent’s potential (as 
established by the production’s enactment, pattern, language, and thought) and to meet the demands of the 
human-computer activity. When characters exhibit external traits that exceed the necessary requirements of 
the activity, users tend to become annoyed and the action is disrupted. An example of the excess 
demonstration to of external traits (relative to a specific activity) is Microsoft’s “Office Assistants” who 
rock back and forth, wiggle their toes, doze in and out of sleep, interrupt with questions, and otherwise 
distract the user (Head, 1 998). In contrast, when computer-based agents exhibit well-integrated traits that 
are appropriate for the activity, they can elicit empathy and influence responses. 

Level 6: Action — the probable outcome or goal of the production that has been collaboratively 
shaped by the user and the computer. It is the entire set of causally and structurally related materials 
emerging from the enactment, pattern, language, thought, and character, and formulated into an “organic 
whole” with a “beginning,” a “middle,” and an “end.” Well-designed beginnings create the potential for 
action within the production by establishing the “rules.” Well-designed middles consistently obey those 
rules while also creating the potential for future action. And no matter how monumental or trivial the 
representation, well-designed endings always supply the logical and pleasurable conclusion of those 
actions, or the catharsis. Thus, Laurel maintained that at the heart of a dramatic theory of representation is 
the need to make optimal use of all available stimuli in order to portray even the most mundane human- 
computer activity in a way that is satisfying, motivating, and complete. 

Sound’s Role in Developing Dramatic Characters 

In the broader dramatic view of metaphorical design, sound effects and speech are not the same 
single experimental variable. In addition to developing character, concrete sounds enrich the enactment of a 
dramatic representation by supplying volumes of information that listeners easily interpret in order to 
extrapolate important details about the world around them (Harmon, 1988; McAdams, 1993; Perkins, 

1983). Symbolic sounds expand the language of a dramatic representation by condensing complex concepts 
into small auditory chunks. 

Thus, sound effects used in the characterization version in study one may have contributed more to 
the learner’s experience than just character development. In fact, eliminating sound effects in the narrative 
version may have altered that treatment’s dramatic structure in ways that made the learner’s experience 
with it markedly different and confounded interpretation. 

Perhaps the question here should have been whether at the language level learners gain more from 
speech that supplies first-person, primary source information (learner as interpreter, eyewitness) or speech 
that supplies third-person, secondary source information (narrator as interpreter, trusted storyteller). That is, 
is it better to advance the “plot” of instructional software using character speech (dialogue or monologue) 
or narration? Given this change in focus, the researchers decided to make modifications to the information- 
processing model program to resolve what we felt were its shortcomings as a dramatic presentation and to 
retest with a second sample. 

Study Two: Monologue versus Narration 

Method 

Sample 

The sample for the study consisted of sixteen (7 male and 9 female) graduate students enrolled in 
an Introduction to Instructional Design course in the same private northeastern U.S. university. The age 
range of the participants was 23-47 (mean = 33.1). The researchers randomly assigned the participants to 
two treatment groups: monologue or narration, with 8 participants in each group. 
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Instrumentation 

Once again, this study employed a pretreatment questionnaire, a posttest, a posttreatment 
questionnaire, and a retention test one week after the treatment. Participants did not complete a delayed 
questionnaire, however. The researchers added a paper-and-pencil pretest of participants’ prior knowledge 
of the information-processing model. Other than this one paper and pencil instruments, all other measures 
were completed electronically. Unlike the first study, the retention test contained no simple knowledge- 
level content recall items. They were eliminated because course readings on information processing were to 
be completed in the intervening week. The retention test still included items on interface properties and 
items asking learners to consider extensions of the content. 

The posttreatment questionnaire assessed overall design of the software (including such things as 
clarity of instructions, attractiveness of graphics, interest level, quality and usefulness of sounds employed, 
and reflection on how much the program helped in learning), ratings of instructional software in general, 
and expressing feelings toward IPP staff. The test-retest reliability coefficient was +.44 {p = .09) for content 
items (pretest to posttest) and +.64 {p = .007) for interface properties items (posttest to retention test) . The 
coefficient alphas for questionnaire sections amenable to such analysis were +.80 for overall software 
design: and +.61 for general instructional software ratings (both /?<.05). In addition, in order to explore 
further the effect of sound on cognitive processing, the researchers created a subscale from the more global 
overall software design section: evaluation of the use of sounds in the program (alpha = +. 94,/?<.05). 
Unlike the posttreatment questionnaire in the first study, the posttreatment questionnaire in this study did 
not ask participants to express attitudes toward the use of instructional software in teaching. 

Treatments 

Prior to the second study, the two treatment applications were recoded using mFactory’s 
mTropolis^'^ authoring tool. The new, “single application” version of the treatments read information from 
an external text file to determine whether it should play the third-person (narration) or first-person 
(monologue) speech files. Other than “personness” of speech files, treatments were identical. As noted 
earlier, the researchers made additional changes in the software intended to eliminate potentially 
confounding interactions among the other elements of dramatic structure. These changes might be 
categorized as changes to the original production’s enactment, pattern, language, and thought. Each 
grouping of design changes is discussed below. 

The enactment of a dramatic representation involves the appropriate use of multi-sensory 
phenomena to portray the intended action. Paivio (1983, 1986) andPaivio, Philipchalk, and Rowe, (1975) 
found that supplying multiple sensory pathways to information, or “dual coding,” may have an additive 
effect on learning. While the first-person treatment in study one included some sound effects, the causal 
relationships between visual and auditory events may not have been established firmly — potentially 
defeating their dual-coding efficacy. 

For example, on the IPP Package-shipping Center “operations area” screens in the earlier version 
(blocks 2 and 4), learners encountered some events that should have made sound and did (like the 
distracting fly), some events that should have made sound continuously but did so only occasionally (like 
the moving conveyor belt and the falling boxes), and some events that should have made sound but did not 
at all (such as the opening and closing of the storage bins). In the revised version of the treatments, ambient 
concrete sound effects were used more realistically and consistently in order to create a more convincing 
representation and to enhance the dual coding of information. 

Pattern involves arranging a representation’s sensory phenomena into pleasing displays. Maslow 
(1970) argued that aesthetically pleasing stimuli can positively change one’s cognitive processes and 
attitudes and motivate continued involvement in the activity. In contrast, unpleasing stimuli can disrupt 
cognition, damage altitudes, and dissuade persistence (Brown, 1988; Norman, 1986). Unfortunately, 
aesthetic inconsistencies in the original production may have affected learners’ experience negatively. 
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Figure 1. Opening simple line art. 



Figure 2. Animated truck arrival. 



For example, researcher observations during study one indicated that once the pattern for sound 
effects had been established as part of the lesson’s enactment, long periods of silence later in the 
application led some learners to believe that there was something wrong with their computers. The 
redesigned treatment ensured no more than a brief silence (I -3 seconds) whenever the computer was 
presenting information. In addition, the opening sequence was changed from static, realistic line art (see 
Figure I) to a simple audio-visual animation of a truck traveling to and arriving at the IPP Package- 
shipping Center (see Figure 2) in order to set up the cartoon-like, animated feel of later screens. Graphics 
throughout the production were made to look more “dimensionally” realistic. A “reversed” version of the 
IPP Package-shipping center operations area image (from the other side of the room) was added to the 
executive function to surprise learners and enhance the sense of “point of view” (see Figure 3). Animation 
was used to draw learners’ attention to particular characters rather than graying down the rest of the screen. 
Screen elements were slightly rearranged and text was edited to flow better. In addition, original sound files 
were remixed to obtain higher audio quality and to sustain more consistent volumes. These and similar 
minor aesthetic changes to the production were intended to create a more engaging environment. The 
capacity of the symbols used in a production, or its language , has a direct bearing on the representation’s 
potential to communicate thought, character, and action. According to von Bertalanfly (1 965) and Brown 
(1 965), symbolic encoding is particularly useful for the storage and retrieval of information. Salomon 
(1 994) proposed that any stimulus can be a symbol, provided that there are established rules or conventions 
of combining and arranging symbols into schemes. Study one did not employ as complete a vocabulary and 
syntax for the sound effects as it might have. For example, sound effects were associated with some of the 
conceptually relevant IPP operations area events (a fly buzzing sound for distractions, a spray sound for 
encoding, a “hmmm” sound for deeper processing) but not with others (the storage and retrieval and the 
executive functions). The revised version established a syntax of sound effects that accompanied the main 
“plot points” throughout the production from their initial metaphorical representations to their final 
descriptions. Other similar revisions at the language level included using the colors blue and red more 
consistently to represent episodic and semantic memories and adding descriptive titles to explanatory 
screens. These changes were aimed at enriching the production’s potential for thought, character, and 
action. 
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Figure 3. Executive's Point of view. 



Dramatic representation calls for computer-based agents to be portrayed in ways that allow 
learners to infer thought . Eisner (1997) suggested that, because the way one thinks about something is 
mediated by the way it is presented, this portrayal may also play an important role in the way the learner 
thinks about information communicated. To maintain the organic unity of the production, it is therefore 
important that the computer’s representations be appropriate for the action and consistent with the 
established enactment, pattern, and language. Unfortunately, the first version of the treatment portrayed the 
computer’s “thoughts” rather inconsistently. 

For example, in the first version of the treatments, the IPP operations animation (blocks 2 and 4) 
did not include an x-ray machine to represent short-term memory processes (see Figure 4). However, later 
in the production, the screen that described these processes in more detail depicted a machine x-raying 
packages. The revised version of the treatments added the x-ray machine to the IPP operations area 
animation so that the x-ray metaphor’s subsequent use would not be unexpected (non-situated) (see Figure 
5). To further learners’ acceptance of the x-ray metaphor, great care was taken in choosing which images 
would portray x-rayed packages’ contents (a snowflake for impermanence; a stop watch for short duration; 
balloons for limited capacity). Images in the earlier version were consonant with what x-rays might reveal 
in a real package, but not with information-processing content (a hidden bomb for impermanence; a dog 
skeleton for short duration; a fish skeleton for limited capacity). Similar care also was taken in choosing 
sound effects to accompany conceptually relevant events in the production. For example, a “slap” was 
added to the fly buzzing sound in order to more clearly represent selective attention overcoming rather than 
becoming distracted by competing stimuli. Other revisions made to the treatments at the level of thought 
included adding dynamic error messages and “wait” cursors. 




Figure 4. Study 1 Figure 5. Study 2 
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Procedures 

The researchers administered the pretreatment session during the regular class one week before the 
treatment. The researchers explained the purpose and procedures of the study and obtained informed 
consent from all participants. In addition, all participants completed a pretreatment questionnaire and a 
pretest. One week later, all participants went to a university Mac Lab to complete the treatment. The 
researchers had previously installed the narration version of the self-instructional software on 8 PowerMac 
8600 computers on the left side of the lab, and installed the monologue version of the software on 8 
PowerMac 8600 computers on the right side of the lab. Each computer had 32MB of RAM, a 17-inch color 
monitor, and a set of earphones. All participants completed treatment in the lab at the same time. When 
participants completed the program, the electronic versions of the posttest and posttreatment questionnaire 
were automatically launched for them to complete. The time each participant spent on the program and on 
the posttest were recorded electronically. One week later, the participants returned to the same lab and 
completed the electronic retention test. 

Results 

Content 

T-tests for independent samples revealed no significant difference between monologue and 
narration groups in prior knowledge on the pretest [t (14) = .62,/? = .55; narration mean = 5.13; monologue 
mean = 5.63] or in posttreatment content recall [t (14) =1.55,/? = . 14; narration mean = 10.38; monologue 
mean = 9.50]. Concerned that, despite random assignment to treatment, small sample size might mask 
differences between groups, the researchers ran an ANCOVA using participants’ pretest scores to adjust 
posttest scores for initial differences. The results approached significance, with the narration group favored 
[F(l,14) = 4.63, p = .051; narration adjusted posttest mean = 10.47; monologue adjusted posttest mean = 
9.40]. As noted earlier, recall of content was not measured on the retention test because course readings 
completed between posttest and retention test might influence findings. 

Interface 

T-tests for independent samples revealed no significant difference between monologue and 
narration groups’ recall of interface properties on the posttest [t(14)= -.49,/? = .63; narration mean = 4.13; 
monologue mean = 3.75] or retention test [t(14) = 1 .08,/? = .30; narration mean = 3.88; monologue mean = 
2.63]. Both groups’ recall of interface properties faded during the week from posttest to retention test 
[narration mean 4.13->3.88, monologue mean 3.75->2.63]. Although participants in the narration group 
initially recalled and continued to recall a greater number of interface properties, this difference was not 
statistically significant. 

Ratings and Attitudes 

The two groups did not differ significantly in their rating of instructional software [t(14) = 1 .28, p 
= .22; narration mean = 12; monologue mean =11], nor in their overall evaluation of the design of the 
software [t(14)=-.83,/?=.42; narration mean: 33.63, monologue mean: 31.75]. T-tests for independent 
samples showed that, of the component items included in overall evaluation of the design of the software, 
the difference between groups approached significance on only one item: “The program was boring.” 
Narration participants disagreed with this statement more strongly than did monologue participants 
[narration mean: 4.38; monologue mean: 3.63; t(14)=2.02,/?=.06]. Although the two groups differed in 
evaluation of the sounds used in the program [narration mean: 7.25; monologue mean: 6.50], and 
perception of how much the metaphor helped them understand the content [narration mean: 4.50; 
monologue mean: 4.13], these differences were not significant [t(14)=.67,/? = .51 and t(14)=1.66,/?=. 12, 
respectively]. 

Expressing Feelings Toward IPP Staff 

The IPP staff consisted of five people: Tom, who was responsible for sorting incoming packages; 
Mary, who was responsible for coding packages according to contents; Ann, who was responsible for 
checking packages for correct coding; Fred, who was responsible for putting packages in storage bins; and 
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Tanika, who was responsible for monitoring the operation of the center and helping to keep everyone on 
task. In terms of participant perceptions of knowing the IPP staff, both groups had 5 participants who said 
they knew the staff “not very well” and 3 said “well.” However, the two groups differed in their perception 
of whom they liked best with 2 votes for Tanika, 4 votes for Tom, and 2 votes for Mary in the narration 
group, and 2 votes for Tanika and 6 votes for Fred in the monologue group. No one in either group reported 
liking Ann best. 

Time-to-completion 

In contrast to the first study, time-to-completion did not correlate significantly with any participant 
measures. Modifications to the two versions of the program largely eliminated differences in time-to- 
completion for the two treatments [monologue mean = 1 8, 17 minutes; narration mean = 1 8.45 minutes] and 
what differences did exist were nonsignificant [t(14) = -.25,/? = .81], The modified animation sequences 
were nearly identical in length [block 2: narration = 172,3 seconds, monologue = 174,2 seconds; block 4: 
narration = 25 1 . 1 seconds, monologue = 25 1 .6 seconds]. 

Interpretation 

Both groups appeared to like their treatments and feel they were engaging and well designed. The 
narration group had higher mean performance on every measure in this second study, except prior 
knowledge of content related to information processing. Since these differences failed to reach significance, 
small sample size may have obscured significant differences. 

After adding sound effects to the narrated version and carefully redesigning the two versions to 
increase dramatic unity, consonance of auxiliary metaphors and images, and the match between images and 
point of view, the researchers are hard pressed to discount the apparent difference between groups. That is, 
it looks as if something about the narrated version helped its participants link content, metaphor, and 
sounds. Reinforcing this conclusion is the fact that narration participants assigned higher ratings to the 
extent to which sounds enhanced the metaphor and the extent to which sounds helped them learn, even 
though all sound effects used in the two programs were identical. In short, use of narration seems to have 
bolstered both the perceived and real effectiveness of an otherwise identical treatment. Once again, a 
negative interpretation still contends: It is possible that the results are actually evidence that some negative 
property of monologue suppressed both the perceived and real effectiveness of an otherwise identical 
treatment. 

Discussion 

If the findings of these two studies may be trusted, single-voice narration is as good as, if not 
superior to, multi-voiced monologues for lower level learning of the type provided by this program. That 
has practical implications for developers. In terms of production time, working with a single-voice narrator 
is much less expensive than with multiple voices for multiple characters. Anyone who has worked to 
balance sound levels in post-production or tweaked digitized speech files can attest to the fact that it is 
almost always easier to do this for a single voice than for multiple voices. Using multi-voiced 
characterizations, therefore, may only be warranted in certain circumstances and for certain types of 
learning. 

In fact, the researchers are inclined to conclude that, despite no significant findings in the second 
study, single- voice narration was more effective than multi-voiced characterization/monologue in these two 
studies. We base this conclusion on multiple component results suggesting superiority in retention and 
attitude from study one and clear superiority on all measures in the second study. Since these two studies 
were intended as initial investigations in a research program exploring how sound affects metaphorical 
design, below we look at how the present findings as a set may be explained and what implications for 
future design and testing might be. 

Explicitness 

The narration script was more explicit than the characterized/monologue script. Learners in the 
narration version were told exactly what they were observing in the initial animation (block 2) and exactly 
what each action in the animation represented in terms of information processing (block 4), In contrast, in 
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the characterized/monologue version, learners had to infer or derive exact relations to content from what 
characters said or thought. Explicit instruction may direct learners more specifically to the key concepts 
that designers intend to teach, increasing the likelihood that learners will acquire key information (Gagne & 
Dricoll, 1988; Glover, Ronning, & Bruning, 1990; Gunter, Estes, & Schwab, 1990; Joyce, Weil, & 

Showers, 1992; Rosenshine & Stevens, 1986). Howell, Fox, and Morehead (1993) recommended that when 
learners have little prior knowledge or little familiarity with the content they are learning, explicit 
instruction is preferred. Participants in the first study were enrolled in a course offering little prior exposure 
to the content, leading the researchers to assume little prior content knowledge (an assumption reinforced 
by participant comments). Similarly, participants in the second study demonstrated lower levels of content 
knowledge on the pretest. Thus, explicit instruction, as employed in the narration version, may have been 
better suited to their learning needs. One way to confirm explicitness’s influence would be to make the 
characterized/monologue version more explicit and the narrated version less explicit and see how findings 
are affected. Another approach would be to work to eliminate any difference in explicitness between 
treatments through detailed content analysis and reiterative pilot testing and see if this eliminated the 
advantage narration demonstrated here. 

Habituation 

Habituation refers to learners becoming accustomed to certain ways of learning. The literature and 
an examination of commercial products demonstrate that the bulk of software using digitized speech does 
so in the form of narration. It may be that the use of multi -voiced characterization or monologues was too 
new or unusual and that novelty effects here acted as disruption effects. If this is the case, the shortness of 
the treatments did not provide sufficient time for such disruption to fade as learners grew accustomed to the 
multi-voiced approach. Future studies might explore how longer exposure or multiple exposure to multi- 
voiced monologues over a longer period of time might affect recall, retention, and attitude. 

Learning Level 

Content recall in these two studies was predominantly on the knowledge level. Results might 
differ if participants were asked to work with higher level learning. Future studies should examine the 
relationship between the demands of the learning task and drama’s role. 

Metaphor Selection 

A metaphorical graphical user interface employs both an underlying metaphor and auxiliary 
metaphors. The package-shipping center metaphor and its accompanying auxiliary metaphors may not have 
been as effective as other possible metaphors and may have confounded the results in some way. Future 
studies will wish to explore participants’ reactions to the metaphors employed in more detail, perhaps using 
qualitative methodology (such as thinkaloud protocols based on a fixed question set). In addition, further 
research may wish to take the identical content and deliver it through a different set of metaphors to see if 
this has any effect on the dependent variables. 

Voice Selection 

Both treatments used the same voice files. The narration was done by one of the researchers, an 
experienced professional with years of recording experience. The other voices were contributed by friends, 
students, and family members. While those voices seemed good and no participant responses suggest that 
they did not like the voices, it is possible that participants found it easier to listen to and comprehend the 
narrator’s voice. One way to test this would be to change the narrator’s voice while leaving all other voices 
the same and see if this affects student performance and attitudes. 

The narrator was male, while 3 of the 5 center staff characters were female. This meant that only 
32.3% of the time in blocks 2 and 4 was spent listening to male monologues, while 67.8% of that time was 
spent listening to female monologues. Future studies might wish to explore how using a female narrator or 
a different balance of male and female characters might change findings. 

Character Development 

Writing scripts for dramatic characters can be very difficult. It is possible that subtle differences in 
wording might make a difference. More detailed analysis of scripts and consultation with experts in 
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dramatic presentation might enhance future versions of these scripts. Similarly, the 
characterized/monologue version relied heavily upon what the characters said; it employed none of the 
other literary character-development techniques discussed earlier. Future research might explore how more 
interactive dramatic approaches (dialogues, thoughts and actions of other characters) might affect 
dependent variables. 

Organic Unity 

Dietrich and Sundell (1974) argued that neither the narrative nor the dramatic methods for 
advancing action through speech is better than the other artistically. What matters is how effectively the 
type of speech chosen contributes to the overall effort to shape the production into a unified, organic whole. 

Both versions of the program employed a traditional “boxes-and-buttons” interface with 
elaboration screens consisting of text overlays controlled by user actions. Of treatment time, in study 1 
approximately half (52.7% narration, 47.3% characterized/monologue) and in study 2 around 60% (60.9% 
narration, 61.8% characterized/monologue) consisted of learners working through such screens. These 
screens may be viewed primarily as textual forms of narration. Thus, in the narration treatment, there was a 
consistency across all screens in the program, since all consisted of some form of narration. In contrast, in 
the characterized/narration treatment, half of the treatment used one dramatic approach while the other half 
used a different approach. This may have acted to break the mimesis, causing learners to view monologues 
as a kind of distracting interlude. One way to test this would be for all screens in the 
characterized/monologue treatment to employ monologues with all information presented auditorial ly by an 
appropriate character from the earlier animation. 

Population Variables 

Both studies worked with adults in education courses. The first study worked with a blend of 
undergraduates and graduates, while the second worked only with graduate students in the first half of an 
introduction to instructional design course. It is possible that findings would differ for other populations, 
particularly younger ones. Additional research is needed to explore how age/level affects ability to work 
with both metaphorical programs and drama-based programs. 
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Abstract 

A recent study comparing the design, development, and performance of metaphorical and non- 
metaphorical versions of a computer-based instructional (CBI) program on stress management for 
undergraduates revealed no significant difference between treatment groups in terms of 
effectiveness of instruction, but marked differences in use of development resources. The 
metaphorical version took two and three quarters as long to develop and cost the developer four 
and a half times as much as its non-metaphorical counterpart. This article examines how 
developers can improve return-on-investment (ROI) for metaphorical interfaces by making key 
instructional design and development decisions. It closes with a set of rules for developers for 
when to select a metaphorical interface; how to reduce coding costs; how to match instructional 
goal, design, and coding approach; and how to reduce the cost of using sound. 

Metaphorical thinking and expression are pervasive parts of our lives. Lakoff and Johnson (1980) 
suggested that the metaphors we use offer clues to the sets of assumptions and comparisons we bring to 
bear. Such assumptions and comparisons, when shared and well understood, allow us to employ a type of 
cognitive shorthand, making messages communicate more than simply what is explicitly stated or shown. 

Observing the power of metaphorical thinking, interface designers have begun to explore ways to 
use metaphor to enhance what their interfaces, and the programs using those interfaces, provide 
(Bielenberg, 1993; Chiou. 1992; Erickson; 1990). Visual metaphors in the graphical user interface (GUI) 
appear to share the same properties as the ones we use in speech (Dent & Rosenberg, 1990), and may help 
learners develop mental models and associate new information with existing understandings (Barker, 1992; 
Jonassen & Hannum, 1987; Jones & Okey, 1995; Marchionini, 1991 ; Mountford, 1990). Metaphorical 
graphical user interfaces (MGUls) seek to extend visual metaphors, creating engaging environments where 
learners interact with complementary content, navigation, learner support, and feedback. By placing 
instruction within a familiar metaphorical framework, designers can offer a consistent visual and 
conceptual structure that not only may help learners build a mental model of the system, but also may offer 
them a safe, familiar environment in which to explore the content of instruction (Jih & Reeves, 1 992). 

Cates (1994, p. 3) divided interface metaphors into two classes: underlying (or primary) and 
auxiliary (or secondary). He defined an underlying metaphor as the principal or first metaphor employed, 
and an auxiliary metaphor as a subsequent metaphor employed by the product. All design elements are 
presented in the context of the underlying metaphor, seeking to use the more familiar metaphorical 
framework to help learners acquire less familiar subject matter (Berkley & Cates, 1996; Bishop & Cates, 
1996; Cates, 1994, 1996; Ohl & Cates, 1997). Examples of MGUls include a car’s dashboard and controls 
(for learning about the capabilities of interactive digital video; Bielenberg, 1993); an old Victorian house 
(to teach about insects; Gay & Mazur, 1991); an office and laboratory (for learning about environmental 
literacy; Affolter, Lo, & Reeves, 1998); and a reporter’s office and mail-order courses (to teach thinking 
skills; Cates, Bishop, Fontana, & vanNest, 1998). 

Metaphorical Interfaces and Learning 

Metaphorical interfaces may enhance learning because the metaphorical environment can supply 
information through multiple pathways: visual, verbal, and auditory. Dual coding theory, developed and 
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tested by Paivio (1971, 1986), assumes that people can encode information in two memory systems, one 
called the semantic processing system, for verbal language, and the other called the imaginic processing 
system, for sensory images. Lachman, Lachman, and Butterfield (1979) cited studies indicating that 
recognition memory for pictures is better than memory for words, which implies that they are represented 
differently. MGUIs — and their accompanying sounds — offer opportunities for learners to establish dual 
pathways and utilize imaginic processing. 

While metaphorical interfaces may help users understand how a program works, it is unclear, 
however, whether their use has any effect on learners’ recall of content. Further, there appear to be no data 
on whether the effort involved in creating a metaphorical interface provides an adequate retum-on- 
investment, given the cost of up-front analysis of both content and metaphor, in addition to instructional 
design, coding, and production costs. 

In a recent study we sought to determine the extent to which an MGUI could enhance content 
recall and retention by undergraduates completing a computer-based instructional program on stress 
management. That study involved the design, development, and empirical comparison of two versions of a 
computer-based instructional (CBI) program: one delivering the content through an MGUI and one 
delivering the content through a non-metaphorical GUI (referred to hereafter as NGUI). Both versions were 
based on researched principles of instructional and interface design (Berkley & Cates, 1998). The results 
suggest that designers considering using metaphorical interfaces need to weigh potential gains against 
measurable costs. 

Design of the Software 

Main Concept 

The CBI program Coping Skills is an 1 8- to 25-minute lesson on stress and the role of sleep in 
managing stress. It is the first module of a series, planned to cover the roles of nutrition and exercise in 
stress management. The MGUI version of the program features a construction-site setting as its underlying 
metaphor; workers lay the foundation of a building. Learners assume the role of intern, helping 
construction-worker characters build a foundation strong enough to withstand the stress load of a soon-to- 
be built university student center. They also help out in the construction-site office. Learners receive 
directions and guidance from construction-site characters, and in turn are asked to use their newly acquired 
knowledge from the program to help one stressed character adopt more healthy coping behaviors. 

The MGUI version of Coping Skills was developed to be an engaging, immersive metaphorical 
environment that uses the construction site as a metaphorical locale for the exploration of stress and its 
stress management. Participants travel not only from one side of the screen to another, but also forward 
through time and space in the course of a virtual morning. A metaphorical environment allows users to 
manipulate objects on the screen, hear appropriate sounds that result from their actions, and "move" on the 
screen (Bishop & Cates, 1996; Cates, 1996). For example, before the program introduces users to the main 
concept, the importance of sleep in withstanding stress, the program illustrates its metaphorical analogue: 
the importance of driving a foundation’s piles to bedrock to anchor the building’s foundation so firmly that 
the building won't collapse over time. With guidance from the character, Len D. Hand, learners participate 
in the metaphorical illustration by dragging and clicking elements on the screen. The pile-driving screen for 
the MGUI version is shown in Figure 1 . 
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Figure 1 

In comparison to the MGUI version, the NGUI version is decontextualized. It offers comparable 
interactivity and content, though within a more traditional "buttons-and-boxes" interface. The NGUI uses 
similar characters who, like their metaphorical equivalents, give learners direction and support. This 
version provides the same number of opportunities for learners to manipulate screen objects and hear 
accompanying sounds. Where the MGUI contains metaphorical objects and sounds related to the 
metaphorical object, the NGUI contains non-metaphorical, representative objects and symbolic sounds. For 
example, the NGUI's introductory game-show exercise (operating much like the memory game. 
Concentration'*’’^) does not use the game show itself as a metaphor for the content. Rather, it serves as a 
temporary framework for showing symbolic or representative images of stress and sleep, allowing learners 
to play a game as a kind of warm-up exercise, and introducing characters who play key roles in guiding and 
presenting the content. The game and the NGUI version of Len D. Hand are shown in Figure 2. 




Fieure 2 
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Elaboration and Practice 

The program subsequently reinforces the main concept with two animated dream sequences, 
triggered by Len dozing off The first dream provides an illustration of the effect of sleep, and the lack of it, 
on college-related stressors — stressors not just academic, but also social, physiological, and psychological 
in nature. The second dream expands upon the main concept by introducing new information: stress and 
sleep have a physiological connection that can affect our biochemical systems. Once again, the MGUI 
versions of the dreams reinforce the "firm foundation" and "bedrock" auxiliary metaphors (see Figures 3 
and 5), while the NGUI version uses the more traditional representational style (see Figures 4 and 6), 




Fisure 5. Fi 2 ure 6. 



Following the dream sequences, both versions of the program offer more detailed information on 
the relationship of stress and sleep to the biochemical systems of blood pressure, metabolism, muscle 
tension, breathing rate, and heart rate. Each interface maintains its structure and approach as it provides 
learners with this elaboration. For example, in the MGUI version, Len turns the "interns" over to another 
character, Ike N. Cope, who assigns learners the task of reviewing five blueprints (tied to the five 
biochemical systems) and asks them to straighten up the office by returning to the appropriate shelves items 
left by others. In this way, users learn about the five subconcepts (blueprints) and receive practice recalling 
the content they have just reviewed (shelving items). 

The NGUI method for presenting subconcepts is simpler, the character of Ike N. Cope appears on 
a "menu" screen featuring five buttons for each of the five subconcepts. He directs users to click on each of 
five buttons to learn more about how stress and sleep affect our biochemical systems. He then exits the 
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screen, and users make their first choice. Figure 7 illustrates text and animation on the MGUI blueprint for 
elaboration on muscle tension, while Figure 8 shows the NGUI version of that elaboration. 





Fi£ure 8. 



Similarly, Figure 9 shows the MGUI embedded practice screen that employs the auxiliary 
metaphor of cleaning up the coffee break area, while Figure 10 shows the comparable non-metaphorical 
version of this activity used in the NGUI. Once learners view the information on biochemical systems and 
complete their review of all eight embedded items, they have completed the module. 




Figure 9. 




Figure 10. 



Equivalency of Versions 

Much care was taken in the design of the MGUI and NGUI versions to ensure that both the 
metaphorical interface and its decontextualized alternative would supply comparable levels of educational 
value and interest. We employed the same general instructional design, used identical content text, and 
made all digitized resources (sound and graphics) as similar as possible. Both versions of Coping Skills 
were approved by expert reviewers for content equivalency and for near-equivalency of instructional 
design, interface design, and learner appeal. 

Study Procedures 

The study utilized a posttest-only control group design with random assignment to treatment and a 
repeated measure for retention of content. Our research sample consisted of 76 volunteer participants at a 
small co-ed state university in Pennsylvania during fall 1998. Forty-three participants completed the 
metaphorical treatment, while 33 completed the non-metaphorical version. Participants completed the 
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initial treatments in a university Macintosh lab in six 90-minute sessions held over a three-day period. The 
lab houses 25 Macintosh Performa'**'^ 6500s, with 32 MB of RAM, CD-ROM drives, attached Zip'*’*^ drives, 
and sound cards. The lab was arranged so that MGUI and NGUI participants were separated, avoiding the 
possible distraction of adjacent participants having screens that differed. Similarly, the use of earphones 
presented program sounds from distracting adjacent participants. 

Content recall (immediately following treatment) was measured using a 24-item posttest. In 
addition, all participants completed a pre- and post-treatment questionnaire. Both versions of the program 
recorded an audit trail to track participants* actions, including time-to-completion. This provided measures 
of learner efficiency. Content retention was measured one week later when participants returned to the 
Macintosh lab to complete the same 24-item posttest, this time with items re-randomized to control for 
order effects. This follow-up treatment was held in six 90-minute sessions over a three-day period. 

Findings 

As anticipated, data analysis revealed no significant difference between treatment groups for 
immediate recall, /(74) = - 1 . 1 0, p = .274 [M mgui = 19.91 (of 24 items), SD = 2. 1 9; Af ngui = 20.45, SD = 
2.1 1]. Contrary to our expectations, however, there was no significant difference in retention test 
performance between treatments, /(74) = -.80,/? = .428 (Af mgui = 19.26, SD = 2.47; M ngui = 19.70, SD = 
2.32). In addition, both treatment groups reported comparable affective reactions to the two versions, as 
shown in Table 1 . The post-treatment questionnaire collecting participant opinions used a five-point Likert 
scale, with 5 representing "strongly agree." 



Table L Measures of Central Tendency. Variability, and Significance for Opinion Statements 



Opinion 


Central Tendency and Variability 


t Value 


P 


The program motivated me. 


A/ MGUI = 3.98, SD = .74 
A/ ngui = 4.06, SD = .67 


/(73) = -.52 


.601 


I found the program interesting. 


^ MGUI “ 4.33, SD — .68 
^ NGUI “ 4.44, SD — .67 


/(73) = .19 


.849 


The program moved from screen 
to screen at a comfortable pace. 


A/ mgui =4.37,5D = .66 
A/ngui =4.13, 5D= 1.13 


/(73)= 1.11 


.274 


I would recommend this program 
to college students. 


A<f mgui = 4.16, 5D = .84 
A^ngui = 4.25, 5*D = .84 


,(73) = -.44 


.659 



Since both the MGUI and NGUI versions of this program were designed in accordance with established 
principles of instructional and interface design, and since the study's participants not only performed 
comparably well but appear to have enjoyed both versions equivalently, this study appears to confirm 
Clark's argument that well-designed instruction is the key to effectiveness (1983, 1 994). 

MGUI vs. NGUI: Efficiency Issues 

In many ways designing an instructional MGUI is the same as designing its non-metaphorical 
alternative. Both versions require the same steps in the instructional design process: identifying goals, 
analyzing the audience and content, determining objectives, developing test items, selecting instructional 
materials for delivery, and storyboarding the product screen by screen. In the development stage, both 
versions of Coping Skills required roughly the same number of animations, sound effects, voice-overs, and 
interactions. 
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The MGUI required a larger effort for two reasons. First, the act of integrating a metaphorical 
interface demanded an extra layer of analysis and design to ensure that our chosen primary and auxiliary 
metaphors supported, rather than confounded, the instructional content. We began with Cates' (1994) 
POPIT analysis method to identify properties, operations, phrases, images, and types related to the 
construction site and office. We followed with a similar analysis of the topic (stress and its management) to 
draw the tightest possible connection between Paivio's ( 1 979 metaphorical topic and vehicle (the 
construction site and office). This procedure identified appropriate semantic, visual, and auditory design 
elements. Additional time was spent researching and acquiring the resources for developing those design 
elements. 

Second, the metaphorical environment required more design and development time because it 
employed extra screens. Both MGUI and NGUI program screens can be divided into two groups: those 
whose primary purpose is to deliver content and those whose primary purpose is not. The latter could be 
called manipulation/transition screens. Manipulation/transition screens serve one of two primary purposes: 
either to provide a manipulative experience to set up the actual content, or to move the user from one visual 
location to another. While both treatments contain an equal number of content screens {n - 16), the MGUI 
treatment contains 1 0 more manipulation/transition screens than the NGUI treatment does -almost twice as 
many, as shown in Table 2. Three of these 10 extra MGUI manipulation/transition screens develop the 
metaphorical pile-driving scenario, while the other seven support the metaphorical office scenario in which 
objects must be dragged to their respective shelves before participants can respond to embedded items. 



Table 2. Number and Percentage of Screens for Each Treatment 



Kind of Screen 


UGUln 


% 


NGUIw 


% 


Manipulation/transition 


21 


57 


11 


41 


Content 


16 


43 


16 


59 


Total 


37 




27 





Those ten extra manipulation/transition screens meant investing extra thought, time, and effort in designing 
storyboards, finding or creating appropriate visual and auditory resources, and then programming those 
manipulations and transitions. 

How did these additional investments translate to development costs? To determine this, we 
analyzed the work logs of the five development staff and two designers involved in creating the two 
equivalent versions of Coping Skills, That analysis revealed that, while the two versions appear to be 
equivalently effective, they represent markedly different investments of time and money. Table 3 details 
what we found. 



Table 3. Total Development Fisures for the Two Versions of Coy ins Skills 



Development Activity 




MGUI 






NGUI 




Hours 


Dollars 


Proportion 


Hours 


Dollars 


Proportion 


Design 


280 


6404 


21.9% 


200 


457 


43.1% 


Development 


876 


20034 


68.7% 


224 


5123 


48.3% 


Production/Debugging 


120 


2744 


9.4% 


40 


915 


8.6% 


Total 


1276 


29182 


100% 


464 


6495 


100% 



Note: All dollar figures shown are actual salary plus benefits for a small development house located in the 
northeastern United States. 



The data in Table 3 show that the MGUI took a 2.75 times as long and cost the developer 4.5 
times as much to bring to completion as the NGUI. While using developer salary (plus benefits) disguises 
the possibility that one might be able to charge a client substantially more for an MGUI product, the earlier 
data on effectiveness suggest that such a client investment may not be warranted. This led us to consider 
how one might improve the retum-on-investment (ROI) for MGUI products. Retum-on-investment is a 
financial measurement expressing the ratio of net benefits to investment, or costs (Long, 1999). For this 
project, retum-on-investment was viewed as the ratio of the effectiveness of the learning experience 
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measured in performance (benefits) to the time, effort, and costs necessary to produce the results 
(investment). 

Improving Development ROI 

Metaphors in the interface seem to work well as organizational and navigational devices. The 
results of this study imply that they also work well to enhance retention of content, but apparently no better 
than a more traditional alternative. Could metaphorical interfaces do more to aid learning without requiring 
even greater developer investment? Despite the findings of our study, metaphors may still enable learners 
to code new knowledge into long term memory; they may still serve to support and scaffold learners as 
they build new or enhance existing schemata. In other words, the power of metaphorical interfaces may still 
be unrealized. But how are instructional designers and CBI developers to unleash that power without a 
disproportionate investment of time and effort? In the course of developing both MGUI and NGUI 
interfaces, we learned some lessons that might serve to guide future metaphorical interface development. 
We propose these below as a series of rules for developers. 

Rule 1: Length of treatment makes a difference. 

Make the learning experience sustained enough. If the instruction is too short, too great a 
proportion of it will be groundwork required to set up the metaphor. Such an investment in building a 
foundation is only likely to be paid back (amortized) over a sizeable chunk of treatment. We strongly 
suspect that this was the case with the MGUI version of Coping Skills. It took the average participant in the 
study 24.5 minutes to complete the MGUI treatment. Of this time, an average of 9.9 minutes (about 40%) 
was spent on manipulation/transition screens, rather than content screens. In contrast, it took the average 
NGUI participant 1 8.3 minutes to complete treatment, with an average of 5.6 minutes (about 30%) spent on 
interface screens. 

It is likely that such an investment in building a metaphorical "foundation" would be better 
realized over a longer treatment. This means that metaphorical design is probably ill suited to treatments 
less than an hour in length, and might actually only come into its ovm after several hours of accumulated 
exposure. Paivio's dual coding traces (1986) may need exercise to be become fully automated. A short 
treatment does not seem able to accomplish this. If the treatment is designed to take less than an hour, a 
traditional NGUI buttons-and-boxes interface should prove more efficient. 

Rule 2: Level of learning matters. 

In order to assess your ultimate ROI, know what return you expect. A detailed determination of 
the program's instructional goals should supply the answer. Coping Skills had a fairly simple instructional 
goal: to enhance recall of content. If low-level learning is sufficient to achieve your instructional goal, 
development of a metaphorical interface may call for too heavy an investment of time and resources for the 
likely return. If processes involving complex understandings are the desired outcome, a metaphor, with its 
possibilities for complex schema building or enhancement, may be able to provide deeper 
conceptualizations and representations than a non-metaphorical interface. 

Rule 3: Let your instructional design make demands on your interface, not vice-versa. 

Norman (1986) described the human-computer interface as a bridge between the physical 
computer system and the goals of the user. The purpose of the interface in a CBI lesson is to enhance 
learning, to help the learner meet the instructional goal. Our findings confirm that instructional design 
appears more important than interface cleverness in creating effective learning systems. Instructional 
design is the most important tool we utilize in designing effective learning systems; interface cleverness 
arguably runs second. If one must lead, let it be the instructional goal. 

Rule 4: Choose auxiliary metaphors carefully and sparingly. 

Don't be so wedded to your metaphorical environment that you skew your ROI. When you choose 
an auxiliary metaphor, think about costs as well as appeal. There are many possible auxiliary metaphors for 
any one underlying metaphor. Some cost a lot less to design and code than others. 
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In designing our metaphorical environment, we thought it important to develop the environment in 
a "cinematic" style; that is, one in which the learner views the screen like an interactive movie. Creating 
that cinematic, immersive environment, however, led to a heavy investment in design and programming in 
order to match every element of content, support, navigation, and interaction with appropriate primary and 
auxiliary metaphors. We spent a lot of time figuring out how to create images and metaphors suited to 
construction. After much effort, this produced clever blueprints for allowing the learner to work with the 
very same content as delivered on a plain buttons-and-boxes screen in the NGUI. The NGUI screen took a 
day to complete, while the equivalent MGUI screen took a week. Ultimately, one must strike a balance 
between instructional effect and cost of development. In selecting auxiliary metaphors, therefore, make 
sure they're highly salient so they contribute not only to metaphorical fidelity but also to instructional 
effectiveness. 

Rule 5: Keep embedded items simple and clean 

Simplicity in embedded items may be more important than metaphorical fidelity or cleverness. 

Our audit trail analyses indicated that approximately 7.3% of learner errors were made as learners dragged 
and dropped objects on the MGUI manipulation/transition screens that preceded the item screens, and on 
the item screens themselves. In contrast, though NGUI participants also dragged and dropped objects on 
their corresponding item screens, the NGUI interface was simpler in its design, leading to no errors 
attributed to difficulty with the drag-and-drop function. As Sweller (1989) argued, when the interface 
makes cognitive demands on the learner, it reduces the amount of processing resources available for 
completion of the task. Certainly a developer wishes the learner to have the maximum amount of 
processing available when learners are asked to respond to items. It may be that having them drag and drop 
objects to a variety of locations on metaphorically complex screens (see for example, Figure 9) consumes 
more cognitive resources than does a simpler screen (see for example, Figure 10). 

Rule 6: Reducing coding costs increases ROL 

Always look for ways to reduce the cost of coding. We suggest that the following points for 
anyone interested in creating an immersive learning experience for CBI, whether it contains a metaphorical 
interface or not. 

Avoid pans; use stills whenever possible. We spent a lot of time panning from one side of a 
metaphorical image to another in the MGUI as a way of helping learners make transitions from one 
"section" of the metaphorical construction site to the next. Pans take a lot of time to do well and the more 
complex the screen is, the more opportunity there is for the programmer to miss something. Also, pans 
make demands on computer CPUs. This means that a program with pans takes more memory and a faster 
processor in order to avoid jerkiness or making the learner wait. What is gained if your beautifully executed 
pan sequence runs poorly or the learner loses interest while waiting for it to finish? 

Avoid complex manipulations. The pile driver screen in our MGUI version took over 200 hours 
(20%) of all the coding and debugging for the MGUI. Of all the errors in the MGUI version, almost 30% 
occurred on the pile-driver screen. It is worth noting that a novice on a construction site would not be asked 
to operate a real pile driver. If we had coded this as a simple automated process with a "computerized" 
control board where the learner clicked to drive the piles, it would have been a slightly more sophisticated 
type of buttons-and-boxes screen and would have been much easier to code, and likely to produce few 
learner errors. By our estimates, this would have saved us over 100 hours and probably $2,500. So, it 
appears that we got a bit too clever for our own good. In fact, based on the number of learner errors on this 
screen (20 of 43 learners in the MGUI treatment made one or more error on this screen), we probably 
reduced return while raising investment. That's bad economics. 

Rule 7: Plan carefully for the use of sound. 

Producing sound is incredibly time-consuming. If you plan to use sound, have a carefully thought- 
out program that does so to maximum effect. Gratuitous or simply clever use of sound may not be worth 
the investment. Recent studies suggest that single-voice narration may be just as effective as individual 
voice characterization (Cates, Bishop, & Hung, 2000) and on-screen speech balloons may be as effective as 
individual voice characterizations (Cates & Bender, 2000). Both versions of Coping Skills made extensive 
use of voice characterization. It takes substantially less time to produce single-voice narration than voice 
characterization, and still less time to produce on-screen speech balloons. 
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Similarly, sound effects can take time to produce and incorporate. We used them extensively in 
the two versions, particularly in the MGUI. If a sound is to act as one of Paivio's dual-coding retrieval 
pathways, it needs to be carefully selected and systematically used. We attempted to do this in the MGUI 
with the ”pile-driven-to-bedrock" and the "pile-not-quite-driven-to-bedrock" sounds. It appears that this 
was not, in and of itself, a sufficiently systematic plan. Bishop (November, 1999) and Hung (April, 1998) 
argued that for sounds to function as schematic pathways, they must be tightly related to the content under 
study and must be repeated sufficiently to permit the pathway to become well established. As noted earlier, 
this may call for more sophisticated conceptions of how sound relates to the instructional goal, and may be 
better suited to programs seeking to help learners acquire and exercise more complex understandings. 

Final Thoughts 

While the data here might indicate that developers should avoid MGUIs, we would like to suggest 
that they not give up too soon. While we support the notion that one should not design products in ways 
that make them more expensive than their effectiveness warrants, we believe that MGUIs have a proper 
place in interface and instructional design. The key is to use them carefully and when they are most likely 
to have the desired effect. 
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Abstract 



We report on a project in which two classes in College Station, Texas conducted school activities 
over a year via videoconference with two classes in Mexico City. We explored the effectiveness of 
distance learning for international team teaching and the ways that students learn through shared 
artwork. In this arts-based case study we asked — What cultural differences manifest themselves 
through the children's murals shared over geographic distances? The activities in the partnership' 
were designed to help the students understand each other's cultures. This study demonstrates that 
students can share artifacts and then interpret them to learn about cultures that they manifest. 
They can gain insight regarding each other's similarities and differences by participating in 
telecommunications partnerships. 

Background and Theoretical Perspective 

We propose that distance technologies can be used to achieve the desirable multicultural ”End(s) 
of Education" that Postman (1995) describes— "to find and promote large, inclusive narratives for all 
students to believe in" (p. 144), and "to help the young transcend individual identity by finding inspiration 
in a story of humanity" (p. 171). By exchanging ideas about content areas being studied, students can 
acquire a broad and inclusive world-view, thereby approaching world citizenship. The primary competency 
of world citizenship is the ability to look at and approach problems as a member of a global society (Parker, 
Ninomiya, & Cogan, 1999). Other competencies include cooperation, responsibility, tolerance, 
appreciation, critical thinking, ability to resolve conflict, political participation, behavioral change to 
protect the environment, and sensitivity to human rights. 

Five human narratives named by Postman (1995) and six ethical questions identified by a 
multinational panel (Parker et al., 1999) provide a framework for a curriculum that addresses multicultural 
understanding and world citizenship. Postman’s narratives are 1) stewardship of the Earth, 2) spirituality, 3) 
democracy, 4) diversity, and 5) the power of language. The six ethical questions are: What should be done 
in order to promote equity and fairness within and among societies? What should be the balance between 
the right to privacy and free and open access to information in information-based societies? What should be 
the balance between protecting the environment and meeting human needs? What should be done to cope 
with population growth, genetic engineering, and children of poverty? What should be done to develop 
shared (universal, global) values while respecting local values? What should be done to secure an ethically 
based distribution of power for deciding policy and action on the above issues? (p. 129) 

Tiessen and Ward (1998) claim that “computer-mediated communication technologies can . . . 
directly support educationally valuable activities that would be difficult through traditional media” (p. 175). 
They describe such activities as providing for — (a) many-to-many communication, (b) distributed learning, 
(c) a shift from teacher control to student ownership, (d) a shift from focus on tasks to focus on 
understanding, and (e) shared understanding through collaboratively constructed artifacts. 

Examples of successful telecommunications projects that have facilitated communication among 
multiple individuals for purposes of discussing issues from different perspectives include one in which 
senior citizens corresponded with elementary school students to share their wealth of knowledge that may 
come with age (Brush, 1998). In another (Zeitz & Kueny, 1998), high schoolers in Osaka, Japan discussed 
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the murder of a Japanese exchange student in Louisiana with high schoolers in Iowa. Such projects broaden 
learners’ worldviews and facilitate the goals of education. 

In a previous study we explored the effectiveness of distance learning and multimedia 
technologies in facilitating an expanded learning community among two teachers and their students 
between two geographically separated schools in Texas (Cifiientes & Murphy, in press-b). The teachers 
collaboratively developed curricular activities and identity-forming multicultural activities for their K- 1 2 
students to conduct over the distance. Predominantly Hispanic students in one school on the Texas border 
with Mexico communicated regularly over a school year with diverse students in a partner school hundreds 
of miles to the north. They participated in collaborative activities and shared multimedia files via 
interactive videoconference. Using qualitative research methods, we discovered that the participating 
teachers developed empowering multicultural relationships while their students developed multicultural 
understanding and positive self-concept. Examples of empowerment and positive self-concept included 
raised levels of academic aspirations and heightened poise during public speaking. 



Project Description 

In the current study we applied the Cultural Connections Distance Learning model with an 
international team of teachers (Cifiientes & Murphy, in press-a). The model contains the following steps 
that are fluid and are not necessarily applied in the given order: 

• establish a vision for expanding the learning community; 

• assess needs; 

• identify distant partners; 

• communicate the vision among teachers and administrators across sites; 

• identify existing activities or develop new ones in response to the identified needs; 

• identify existing or develop new professional development workshops needed to support envisioned 
activities; 

• conduct professional development workshops as needed and just-on-time; 

• connect teachers with distant partners who share interests; 

• design, develop, and schedule distance learning activities; 

• collaboratively structure initial meetings and follow-up activities; and plan and implement documentation 
and evaluation strategies. 

• assist students in developing products to share for purposes of distance learning; 

For this project, all four classes attended poetry writing workshops in which they wrote poems 
about the other nation. The need for collaboration was established through analysis of these poems that 
students wrote about each other’s cultures prior to any DL activities. Most Texan children had little 
knowledge of Mexico and had to write about their own culture or an imaginary place, or to refer to 
stereotypes in order to create a poem. On the other hand, Mexican children had enough previous experience 
with U.S culture to be able to describe the culture in poetry accurately (Cifiientes & Murphy, 1999). 

The Texan and Mexican students wrote a story of a day in the life of a fourth-grader in the other 
country. Both of these activities were meant to help students establish how much they did or did not know 
about each other’s lives so that, as the year progressed, they could set personal goals for gaining 
knowledge. In addition, they completed a survey that provided indications of the values and artistic 
interests of each child. They also created student profile sheets that described their families, favorite 
subjects in school, favorite foods, and special interests, and also included their pictures. They shared these 
poems, stories, surveys, and profiles with each other via videoconference at the beginning of the 
partnership; the teachers then made them available during the school year. Because most of these 
documents were not electronic and the mail system was insecure, they used a messenger who traveled 
between Texas and Mexico regularly for unrelated business purposes. 

As in the Texas partnership, several of the curricular activities were stimulated by the book / Felt 
Like I Was From Another Planet (Dresser, 1 994). For example, in a pilot study conducted the previous 
year, students conducted an activity on table manners with the goals of learning about: 
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Cultural adaptations through imitation and trial and error; cultural relativity — not being 
judgmental about other people’s ways; the influence of cultural rules and customs that 
have been informally and unconsciously learned and accepted, yet rarely questioned; and 
what is acceptable and approved in one culture yet might be offensive or unacceptable 
elsewhere (p. 47). 

The participating classes then met twice in 90-minute videoconferences to introduce themselves, 
show collages of images representing their family and interests, and share personal ancestry. After these 
meetings, students were able to discuss their cultural similarities and differences and could recognize the 
differences in levels of knowledge about each other’s cultures discussed earlier in this paper. 

A third 90-minute videoconference was part of a unit designed to help students understand the 
interpretive nature of history and the role that perspective plays in that interpretation. Prior to the 
videoconference, students read a short story written by the first author, “The Battle of Cottonwood Ridge,” 
associated with events of the battle of the Alamo. To encourage objectivity, the names Texas, Mexico, and 
the Alamo were changed. Students pretended to be present at the battle: they picked sides, and wrote a 
diary entry about what they did, why, and what happened to them. At that point, the teachers revealed to 
their students that they had read about the battle of the Alamo of 1 836 and that Texas and Mexico were 
involved. Next, students read the chapters about the period of the Revolution from both the Texas and 
Mexican texts and conducted supportive reading activities. The Mexican text was translated to English for 
the American students. The classes independently identified similarities and differences between the stories 
told in the texts. Then, each student wrote a comparative essay regarding Texan and Mexican 
interpretations of the Texas Revolution. In small groups students developed brief reenactments of a story 
about the Alamo. During the fourth 90-minute videoconference, students alternated across sites to read 
diary entries, perform reenactments, and read comparative essays that they wrote about cross-cultural 
interpretations of the battle of the Alamo. After the videoconference, they discussed what they remembered 
from the diaries, reenactments, and essays, and then they reviewed what was learned about the reasons for 
and events of the battle of the Alamo and the different perspectives of the United States and Mexico. In 
another unit, the classes acted out folktales and sang folksongs with each other via a fourth 90-minute 
videoconference. 

In the mural unit under study, students participated in three activities. First, students collected 
pictures of local murals. The College Station students were delighted to learn that their small town 
contained over 100 murals, of which they saw pictures of 1 5. The Mexican students took a field trip around 
Mexico City to explore the works of mural ists such as Diego Rivera and Orozco. 

The next mural activity was for each of the four classes to create its own mural. Teachers provided 
minimal guidance regarding the content of the murals except to direct the students to represent their culture 
and who they are as a class. The students were in charge of the design and development of their own class 
murals and did not see each other’s murals during this period. Teachers explicitly instructed the students 
not to discuss their murals with students in the other classes prior to sharing them with all four groups. 

The third activity was a videoconference for sharing both artists’ and students’ murals. During the 
videoconference, each student folded lined paper in half to compose lists of images in the murals created by 
artists from College Station and Mexico City; they then compared the types of images used and discussed 
their contents. After this part of the videoconference, each class shared its mural with the other three classes 
and again with folded paper, they composed lists of images in the murals and discussed their contents. 



Methods 

This study is an investigation of the content and style of the Texan and Mexican students' murals. 
Two classes in Mexico City conducted school activities with two classes in College Station, Texas via 
videoconference over a school year. We asked. What cultural differences reveal themselves through the 
children’s art shared over geographic distances? The investigation relied on arts-based case study methods 
(Diamond & Mullen, 1999). 

Participants 

Four teachers, two in College Station, Texas and two in Mexico City, partnered for planning and 
implementation of curricular activities with their students. The 41 fourth-grade students in two classes in 
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College Station and the 47 fourth-grade students in two classes in Mexico City were active in the 
partnership throughout an academic year. Of the Texan students, 17 were male and 24 were female, 26 
were White, 8 were African American, 4 were Hispanic, and 3 were Asian. All but 2 were U.S citizens, 1 
being Indian and another South Korean. Of the Mexican students, 26 were male and 20 were female. All 
were Hispanic Mexican citizens except for 2 Hispanics who were bom in the U.S. 

The Texan school was public. The Mexican school was a private school that met in a converted 
home in an exclusive neighborhood of the city. The school partners in both Texas and Mexico City were 
equipped with multimedia software, Internet connections, and interactive compressed video systems at 
local distance learning centers. The project administrators in both Texas and Mexico helped with planning 
and facilitation of connections. 

Data Sources and Analysis 

We conducted content analyses, as described by Emerson, Fretz, and Shaw (1995), of one of 
murals created by students from each school. During and upon completion of data collection, we used the 
two-phase process of content analysis, open coding and focused coding. During open coding we analyzed 
the murals to identify ideas, themes, or issues they suggested, no matter how varied and disparate. This 
process involved writing initial memos to ourselves (Miles & Huberman, 1994). For instance, initial 
memos indicated that the Texan mural exhibited discrete symbols of Texas and the local university. On the 
other hand, the Mexican mural reflected technological progress and pride in history. After initial memoing, 
we created captions so that as themes, patterns, and variations of particular interest emerged, we could enter 
data under these categories. The categories that emerged were symbols, labels, and artistic approach. For 
focused coding, we examined the murals as well as our memos on a line-by-line basis, giving special 
attention to categories identified during open coding. As a result, we provide rich description of the murals 
unit in order to answer the research question. 

Results 

To address the research question related to cultural differences manifested in Texan and Mexican 
murals, we first describe contents of the murals that the children drew. We then discuss the symbols, labels, 
and artistic approaches conveyed by the students. The section ends with a comparison of the cultures as 
manifested in the murals. 

The Texan mural was drawn with colored felt-tip markers on white butcher paper. The mural was 
framed by the children’s individual, equally spaced handprints labeled with their names. Inside the frame 
was a collection of individual images representing the local culture. Reading from left to right, the images 
included, for instance, a cowboy boot and the elementary school. No people were included in the Texan 
mural. Instead, the images were all Texas symbols: an oil well, the Alamo, a Texas map and flag, and the 
official state symbols, that included a lone star, a mockingbird, a bluebonnet, a longhorn, and an armadillo. 

Eleven specific places in Texas were labeled with their names on the mural including Texas A&M 
University, NASA, and Rock Prairie Elementary School. The children also labeled place symbols such as 
“bluebonnets,” the state flower, and “Bonfire,” a local university tradition. Other labels such as “We love 
Texas!” and “Remember the Alamo!” indicated commitment to place. The artistic approach adopted by the 
Texan students was for each individual to make an independent contribution using the theme of handprints. 
Creation of the frame would have involved students’ consulting with each other about the placement of the 
handprints. Individual images throughout the mural were discrete from each other and while related by 
theme of place, they were not connected graphically to each other in any way. 

The Mexican mural was drawn with brightly colored pastel markers on brown paper and the 
background was filled in with a blue pastel marker. The mural told a story of technological progress of 
humankind from prehistory to the future that read from left to right. Prehistoric images included cavemen 
with spears and a woman weaving. The present day was represented by such symbols as computers and 
kids in baggy pants. The future included an unsmiling alien and black spaceships. All symbols were 
connected to each other both thematically and visually. Images of 35 people in the context of inventions of 
humankind were portrayed in the mural. The past and present were portrayed as friendly while the future 
was portrayed as mostly dark and sinister. 

Labels consisted solely of the following: “Atomic bomb,” “Phone,” and seven entertainment 
names, such as “Spice Girls” and “Titanic.” The Mexican students’ artistic approach reflected consensus 
regarding placement of each image on the mural. Each image had its place in the historic continuum and 
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was connected to the others by the pastel background. To help the story unfold from left to right, the murals 
were dense with imagery. 

The cultural differences manifested through the children’s murals and then shared over geographic 
distances included choices made by the students regarding what to represent and how to convey an artistic 
message. Texans chose not to represent people, while the Mexican mural was replete with humans and 
alien figures. The Texan mural’s theme related to place: school, community, and had local and state appeal 
in its symbols. On the other hand, the Mexican mural portrayed events across time and its theme had 
universal, global appeal. Texans used many labels to convey their message, while Mexicans rarely used 
language in their design. 

The design choices that each class made resulted in murals that clearly conveyed different artistic 
approaches. The Texans took an individualistic approach with each student making a singular contribution 
on the frame and then adding individual disconnected symbols. The Mexicans took a holistic approach with 
each student’s efforts contributing to the flow of the story. 

Educational and Scientific Implications 

This project demonstrates that in networked classrooms students can connect with distant others to 
learn about and from their perspectives. In addition, distance technologies can foster team teaching and 
intercultural relationships across geographical distance. The story of this case of international team 
teaching provides a model for others to expand their learning communities beyond national borders. 
Students can share artifacts and then interpret them to learn about cultures they manifest. They can gain 
insight regarding each other’s similarities and differences by participating in telecommunications 
partnerships. 
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Abstract 

The purpose of this presentation is to discuss a conceptual framework for considering the process 
of developing multiple perspectives in collaborative environments - particularly those that 
emphasize text- or conversation-driven contexts. The theoretical background underlying the 
framework is presented, along with a description of each component of the framework. We 
discuss the importance of collaborative environments for online learning environments, as well as 
implications for design. 

Traditional learning environments, whether they be public education, university education, or 
employee training programs, lean toward the individual as agent, often neglecting the way people employ 
the environment (tools, artifacts, other people) to support, share, and undertake aspects of cognition 
(Perkins, 1993). But, as Roy Pea (1993) states, the mind rarely works alone. One only needs to witness 
events in everyday and professional communities to realize the importance of socially constructed meaning. 
Church groups engage in bible studies, bookstores host numerous book clubs, avid readers form informal 
book clubs, support groups are formed for any number of ailments and tragic experiences, professional 
organizations hold annual conferences where members share knowledge and opinions, involvement in 
professional journals as editors provides the opportunity to read and critique with others, and even the 
process of research lends itself to socially constructed meaning, as researchers consult not only other 
articles, but other researchers. 

In kind, current theoretical perspectives such as cognitive apprenticeship (Collins, Brown, & 
Newman, 1989), knowledge building communities (Scardamalia & Bereiter, 1994), and distributed 
cognition (Pea, 1993; Perkins, 1993; Salomon, 1993) reflect the importance of articulating, comparing, and 
evaluating multiple perspectives. Although the importance of cognition “in solo” is often emphasized in 
schools, the role of social interaction (dialog in particular) in shaping individual and shared points of view 
can also be found in many teaching and learning practices. In literature classrooms, for instance, 
interpreting text is rarely an individual activity; classes often engage in discussion about the author's intent, 
meaning, tone, and mood, as well as about how the author's environment, the political, religious, and socio- 
historical background of the time would likely have affected the literature. Similarly, discussion of music, 
lyrics, and composition within music classrooms can strengthen understanding of the lyrics, or of a style or 
piece. Art history classes engage in discussions about the artists, their lives, and what was occurring in the 
world around them in order for them to conceive of a particular piece. Because each person in the 
classroom has a different knowledge base, different experiences, and different interests, sharing becomes a 
meaningful activity, providing members of the classroom with ideas that they would never have conceived 
individually. In any classroom where new ideas are being introduced, whether through text or lecture, 
discussions play a vital role in developing understanding and considering the complexity of multiple 
perspectives. 

In recent years, much attention has been dedicated towards the design of intentional discourse and 
learning communities that rely upon Internet or network technology (see, for example, Scardamalia, 
Berieter, McLean, Swallow, & Woodruff, 1989; Guzdial, 1997). The information-rich and communicative 
aspects of the Internet might offer a much needed avenue for collaboration among people with alternate 
perspectives by providing environments devoted to supplying learners with information and activities that 
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could cross ethnic, cultural, and socio-economic boundaries and by providing spaces for negotiation. As 
we create more environments to support online collaboration, strategies to support the interpretation of 
multiple perspectives become more and more important. 

As demand for educational communities to incorporate Internet technologies into their instruction 
increases, distance education programs that offer courses solely via the Internet are becoming increasingly 
common. As these environments become more commonplace, instructors and designers must provide 
students with the tools necessary to develop valid ideas and interpretations of material and the means with 
which to develop a shared perspective. Thus, the need to consider how learners develop meaning and 
engage effectively in collaborative environments has become an increasingly important issue to educational 
technologists. In order to establish such an environment, designers must consider the processes underlying 
the development of multiple perspectives and offer environments that engage learners in those processes. 
Accordingly, the purpose of this paper is to provide a general framework of such processes that could be 
used to support the development of collaborative learning environments, particularly those that are 
mediated electronically. 

Theoretical Background 

Traditional education views the learner as a retainer of, rather than a processor of, information 
(Strike & Posner, 1985). Strike and Posner posit that learning is a process of inquiry and results from 
relating what one has encountered to one’s current ideas. To learn, then, one must understand a new idea, 
determine its truth value, and judge its consistency with other ideas. In conceptual change theory, “learners 
incorporate new conceptions into current cognitive structures, and then replace conceptions which have 
become dysfunctional with new ones (1985).” 

The conceptual change epistemology recognizes that knowledge is the product of the interaction 
that takes place during problem solving activities between experience and current conceptions. So, rather 
than simply adding new ideas to old, an interaction takes place that sometimes results in a major alteration 
of old ideas. In order for conceptual change to occur, though, the student must recognize the 
incompatibility of his/her current and new conceptions. Although a solution might have been appropriate 
or feasible in a past experience, the student might recognize its limitations to a new experience. Through 
the process of solving the problem and comparing perspectives, the student will begin to see reasons for the 
limitations and adjust his/her perceptions accordingly. 

While dissatisfaction with an existing conception is a major condition for developing conceptual 
change according to Strike and Posner (1985), other conditions must also exist. Students must understand 
the new conceptions at some level, at least minimally, and find the new conception plausible. For example, 
given the concept "an object in motion stays in motion," a student whose only experience is rolling a ball 
and watching it come to a stop may not believe in the plausibility of the concept because he/she has been 
unable to experience it. The opportunity to experience this concept, at least vicariously through use of tools 
and artifacts, must be afforded for conceptual change to occur. 

Yet another condition for conceptual change is that it expands the student’s understanding of the 
concept; that is, it solves problems more fully. If the students’ background knowledge and prior experience 
is too strong to override the new experiences, they are likely to continue to provide a more viable solution 
for the student. Unless the conception appears fruitful, students will remain committed to prior conceptions 
(Strike and Posner, 1985). 

Convergent Conceptual Change Most disciplines expound the necessity to create instruction that 
fosters problem solving and conceptual development. By their very nature, such activities engender 
divergence of thought. In fact, recent learning theories have replaced the positivist term "interaction" with 
the term "transaction," which implies "unfractured observation" of the whole system (Rosenblatt, 1994). 
Those enamored by this new paradigm see the individual "as part of nature, continuously in transaction 
with the environment (Rosenblatt, 1994)." Recognizing that individuals bring their own experiential 
reservoirs to every learning experience, transactionists recognize the inevitability of multiple 
interpretations. Many researchers have developed theories about conceptual change in collaborative 
learning environments and have conducted studies which indicate that students often form more 
sophisticated understandings of problems, text, and theories collaboratively, rather than “in solo.” Jeremy 
Roschelle (1996) posits that convergence - the construction of shared meanings for conversations, 
concepts, and experiences ~ is the ultimate goal of collaboration. His idea of convergent conceptual change 
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suggests that through conversational interaction, students construct relational meanings to scientific 
concepts, and that these meanings are developed iteratively, so that students collectively display, confirm, 
and repair shared meaning. 

Convergent interpretations of text Recent theories of textual interpretation, such as reader 
response, acknowledge that there are innumerable separate transactions between readers and texts 
(Rosenblatt, 1994). In reader response, reading is described as the creation of meaning rather than the 
discovery of meaning which suggests that the significance of the text does not lie in a specific meaning 
implicit within the text, but instead, in the fact that meaning emerges from the background knowledge and 
experiences previously sealed within the specific reader. Louise Rosenblatt (1994) describes the process of 
reading literature as a ’’two way process involving a reader and a text at a particular time under particular 
circumstances." Reader response, then, can be defined as an approach which assumes that literary meaning 
is a transaction between the reader and the text, where the meaning of literature is not contained in a static 
text, and readers comprehend differently because every reader is unique, so that personal responses to text 
are critical to meaning-making (Flood & Lapp, 1988). At the same time, we must also admit that text can 
be misinterpreted and misrepresented by readers. Readers often bring with them the same type of flawed 
background knowledge as they do when discovering new scientific concepts. Thus, the necessity for 
establishing shared meanings for text becomes important. By engaging in similar processes as those 
described by Roschelle (1996), convergent meanings can also be established when collaboratively 
developing meaning from text, a complex task which calls for the learners to be thoughtful, argumentative, 
attentive, and, perhaps most importantly, intentional. 



A Framework for Collaborative Conceptual Development 

As society continues to place importance on problem-solving skills and teamwork, and as topics 
under study continue to result in a divergence of ideas, the need to consider how people develop meaning in 
collaborative environments is increasingly important. This theoretical framework (see Figure 1) presents a 
method for considering the process of conceptual development in electronically-mediated collaborative 
environments - particularly those that emphasize text - or conversation-driven contexts. 

Figure I. A theoretical framework for conceptual development in collaborative environments 



( 

Meta- 

cognition 



*4 



% 




Underlying this framework is that collaboratively, we build more sophisticated understandings 
than individually (Pea, 1993). Ideally, each participant brings to the collaborative environment some prior 
knowledge, as well as separate and distinctly different personal experiences. Having read the text or 
derived a theory or idea independently, the student develops an initial, quite often naive, perception. The 
learner begins a more sophisticated interpretation process by engaging in activities that expand those initial 
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perceptions, activating prior knowledge, sharing personal experiences, using tools to aid in developing 
meaning, and, perhaps most importantly, finding varied perspectives through artifacts and dynamic 
discussions. 

In collaborative environments, the students collectively develop, negotiate, and evaluate shared 
meaning. Ultimately, though, the goal must be to rectify perspectives and construct coherence individually. 
An iterative process, the interpreter must become an "Integrative Analyst" who deliberately integrates prior 
knowledge, shared cognition, and information exploration to individually elucidate coherence (Chang & 
McDaniel, 1995). By calling upon meta-cognitive strategies, this integrative analyst should begin to 
choose ideas, interpretations, and opinions to ignore, to discard, to elaborate, or to accommodate and 
accept. 

Components of the Framework 

Prior knowledge 

Because learning takes place when connections between concepts are made, a primary goal of 
instruction should be to provide students with opportunities to establish these connections. Activating 
students’ prior knowledge provides an elementary, yet extremely valuable, way to assist establishing 
connections. Without activation of prior knowledge, it is unlikely that students will make successful 
connections. As students integrate new material with pre-existing knowledge, they create links between the 
concepts, constraining the meanings and altering previously established knowledge. 

The importance of making connections to prior knowledge to establish meanings has been well 
documented, as has the fact that students come to the learning experience with naiVe concepts and limited 
background knowledge. Novice-expert studies, for instance, confirm that novice learners often have not 
developed the knowledge structures necessary to negotiate a complex domain, whether strategically, meta- 
cognitively, or conceptually (Gick, 1 986). Novice learners are limited in ability to access and integrate 
with related prior experiences, a critical activity for meaningful learning to occur (Mayer, 1 989). 
Consequently, learners might not see the relevance of other points of view due to limited understanding of 
the topic or limited formulation of their own interpretations. 

Bransford (1984) argues that questions of schema acquisition are just as important as questions of 
schema activation. Bransford expounds that a mismatch between initial interpretations and the phrasing of 
questions can cause considerable deficits in memory performance. Sometimes, perspectives one takes upon 
initial reading of a story, for example, affect the interpretation of the significance of information. Initial 
interpretations can be affected by preconceptions or biases about the material being studied (cultural 
materials, historical references, etc.). In such cases, if readers take a certain slant, questions about aspects 
of the story relevant to an alternate interpretation will likely cause contusion. Thus, Bransford argues that 
we need to confront the problem of helping students develop new schemata or refine old schemata. He 
suggests that one way to accomplish this task might be to give learners information that can clarify the 
significance or relevance of various references within text or within any learning environment. Clearly, 
upon activation of prior knowledge, students begin to make connections between new material and pre- 
existing knowledge. In the event that student’s have weak prior knowledge, provision of prompts and cues 
might become necessary. 

Initial perspectives. 

Most learning is not the acquisition of something that is totally new; learning most often is either 
incorporated within prior knowledge or modifies prior knowledge (Vosniadou, 1 987). As a result, an 
important aspect of the model involves having students articulate their initial perspectives (such as 
generating interpretations or hypotheses about materials). Without articulation of initial perspectives, 
students’ thinking may remain tacit and vague. By asking students to make learning explicit, we help them 
to discover what they think and know about the topic (Schwartz, Lin, Brophy, & Bransford, 1999). Also, 
students will have concrete ideas with which to compare and contrast in future activities that may help in 
later processes. 

Vygotsky (1978) introduced the concept of intersubjectivity, a shared understanding of the goal, 
task, or material. Without this shared understanding, he posits that students will be unable to proceed 
together to establish greater competence. By articulating initial perspectives, students and other members 
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of the learning team can begin to move to'wards the goal of intersubjectivity; thus, allowing for a more 
sophisticated learning to occur. 

Sharing of multiple perspectives. 

Perspectives can be expanded using a multitude of sources. In situations involving making 
meaning from text, for example, perspectives include the author’s perspective, the reader’s perspective, the 
teacher’s perspective, the class’ perspective, the individual student’s perspective, and the critic’s 
perspective (Chase & Hynd, 1987). Yet another perspective is the reader’s changing interpretations when 
faced with challenges to initial interpretations by any of the aforementioned sources (Spiegel, 1 998). For 
this reason, discussion after reading is a valued activity that allows for argumentation, reflection, and social 
negotiation of meaning, activities that help the reader form more sophisticated meanings from the text. 

Observations of related experience 

Although discussion or social interaction may occur solely between one student and teacher, some 
measure of shared meaning is often also established among the whole group. Indeed, Lave and Wenger’s 
notion of legitimate peripheral participation (1991) suggests that learning can occur without active 
participation. Certain members of a class, for instance, may dominate the discussion, based primarily on 
their ownership of ideas, or on a more sophisticated understanding of the topic being discussed. However, 
those students on the periphery, students not actively engaged in the discussion, can begin to formulate 
ideas and develop more sophisticated understanding through another student’s individual discussion with 
the teacher. In literature courses, for example, a student may, because of his/her own experiences, interpret 
a literary work in a completely different way than other students. Like most literature, Theodore Roethke’s 
poem, ”My Papa's Waltz” is vague enough to lend itself to a variety of interpretations. Because the 
narrator reminisces about a specific evening with his father that proved painful to him, some students may 
interpret this poem as describing child abuse. Although this is not the standard interpretation of the poem, 
the discussion between the teacher and student who has interpreted it this way may prove to provide the 
rest of the class with a new interpretation that they must consider and discard or accept. The resulting 
discussion would necessitate that students utilize an evaluative function, in determining whether or not such 
interpretations provide adequate representation of the poem. As they develop this more sophisticated 
understanding by immersing themselves in the reading of text and by listening carefully to discussion, 
students move from the periphery to the center, becoming progressively more engaged in the practice by 
involving themselves more and more in the discussion. 

Related cases 

Whereas the lessons we understand best are those in which we have personally invested effort, 
related cases can provide representations of experiences we have not had (Jonassen, 1 999; Kolodner, 1 999). 
When students encounter problems, they try to find similar cases that provide solutions for those problems. 
Students can compare related cases to their specific problem to ascertain whether the solution, or a similar 
solution, might be feasible in their present situation. Also, since we encode knowledge as stories about 
experiences (Shank, 1990), the use of stories of similar situations can serve as lessons to the students about 
the problem at hand (Jonassen, 1 999). 

An additional aspect of providing related cases to students, according to Jonassen (1999), is to 
provide multiple perspectives or interpretations of problems to avoid the insularity that often occurs as a 
result of traditional instruction, which can filter out the complexity of the problem and result in shallow 
understandings. Cognitive flexibility theory posits that the provision of multiple representations of content 
will disclose the complexity of any knowledge domain (Spiro, Vispoel, Schmitz, Samarapungavan, & 
Boerger, 1 987). By providing related stories and personal experiences, as well as links to similar literature, 
for example, students could make connections vicariously, particularly in situations where they have little 
or no experience with the topics and events in the literature. For instance, when studying Elie Weisel’s 
Night, a story that depicts life in a concentration camp, providing students with access to other stories or to 
individuals who can relate real-life experiences in concentration camps might help solidify their 
understanding of the experience. 
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Transformation 

By employing all aspects of the environments (tools, artifacts, other people) to support, share, and 
undertake aspects of cognition, we can create meaningful learning environments that result in sophisticated 
understanding of complex concepts (Perkins, 1993). Expansion of perspectives can be achieved through a 
multitude of sources, not the least being experimentation tools. Jonassen (1999) argues that learning 
environments should provide opportunities for students to learn with technology. 

Although tools themselves do not enhance cognitive activity, by providing students with tools for 
experimentation, we provide an opportunity to enhance, augment, or extend their thinking (Hannafin, Land, 
& Oliver, 1 999). Hannafin et al. discuss various types of tools that may provide ways to represent abstract 
ideas concretely. Some of these tools aid students in processing information. Included in this category are 
(1) selection tools to support selection of relevant information, (2) collection tools for gathering resources, 
(3) organizing tools for representing relationships among ideas, (4) integrating tools for linking new and 
existing knowledge, and (5) generating tools for creating things. 

More conducive to experimentation are manipulation tools, which aid learners in the processes of 
verifying, testing, and extending understandings (Hannafin et al., 1 999). Students can be encouraged to 
develop initial theories and to use these manipulation tools to validate their ideas. To use manipulation 
tools to effectively restructure knowledge, students must be aware of their existing beliefs. Many 
microworld and simulation tools currently exist with which students can collaboratively test pre-existing 
scientific beliefs. 

Negotiation 

Complex activities, such as critical evaluation or responding to literature, require social 
interaction, as provided in cognitive apprenticeships (Brown, Collins, & Duguid, 1989). Pea (1993) posits 
that communication is a two-way transmission in which meanings begin to emerge between the parties 
involved in the transmission. He suggests that meaningful negotiation is integral to this learning process. 
Negotiation of meaning involves a number of interactional procedures, including gestures, facial 
expressions, requests for clarification or elaboration, commentaries, and linguistic devices that aid in 
signaling and fixing troubles in shared understanding. 

Negotiation environments often facilitate successful completion of tasks because group members 
have different strengths and weaknesses in background knowledge and skills. Negotiation environments 
allow students to utilize the individual strengths of all group members to complete activities. Because they 
would be unable to take advantage of these opportunities, students working independently would likely 
take longer to complete tasks, as well as develop less sophisticated understanding of topic materials. 

Vygotsky (1978) introduced the idea of the zone of proximal development, the difference between 
what a student can accomplish alone and what a student can accomplish with adult guidance or in 
collaboration with more capable peers. Vygotsky believed that social interaction provides growth in human 
competence and that thinking is based on social activity that is internalized. Any individual’s cognitive 
development is dependent on social resources provided within the learning environment. As the novice 
learners become more immersed in the environment, they are guided to greater competence through the 
various resources at their disposal. 

The notion of intersubjectivity, tentatively established earlier in the process, must be made 
concrete during negotiation. Because learners, particularly in heterogeneous learning environments, are at 
different levels of conceptual understanding, intersubjectivity can be hard to accomplish. Nonetheless, 
intersubjectivity must be established for students to develop sophisticated understandings about a topic. 
Without it, learners would be working at cross-purposes. 

Organization and evaluation. 

Because students will be exposed to large quantities of information, they need to think critically 
about the information they have collected during the expansion process. The learner must organize 
selected information in a coherent way, building a text base or a verbal mental model of the situation 
described in the text (Mayer, Steinhoff, Bower, & Mars, 1995). The organization activity takes place in 
working memory, which is constrained by limited resources for processing (Mayer, 1999). A possible 
strategy for organizing information might be for students to begin by summarizing the various pieces of 
information they have collected. After the summarization process, a reasonable step would be to organize 
the information into meaningful units. Once they establish meaningful units, students can begin to 
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synthesize the massive amounts of information, or to begin to search for some confirming or disconfirming 
evidence regarding the information that will help to clarify ideas and positions. In some cases, all these 
steps may not be necessary. Any process that engages students in organizing the information in such a way 
as to make it meaningful will accomplish the goal. 

After the data has been organized, students can begin to evaluation the relevancy, validity, 
accuracy, and reliability of information they locate and receive from various sources. As information 
sources become more easily accessible, students will collect massive amounts of information. By engaging 
in these processes, students set themselves up to be successful in the increasingly important task of 
reflecting on the information and rectifying perspectives based on the information they have acquired. 

Reflection and rectification. 

Although learners often construct mental models based on the integration of background 
knowledge and initial inferences, the importance of reconstructing those models, based on a convergence of 
ideas from socio-cultural experiences and discourse, cannot be discounted. Ultimately, though, the goal 
must be to establish coherence individually. Responsibility lies within the intentionality of the learner - 
his/her intention to reconsider and reconstruct. This process is iterative. The learner must begin and 
intentional process to deliberately integrate prior knowledge, shared cognition, and information exploration 
and coherence (Chang & McDaniel, 1995). By choosing ideas, interpretations, and opinions to ignore, to 
discard, to abandon, or to accommodate and accept, the learner begins the process of reconstruction. But, 
only by returning to the issue when new material presents conflict, and revisiting the process, will 
refinement of individual knowledge continue. 

Meta-cognitive strategies 

Encompassing the entire framework is the necessity of learners’ knowledge and utilization of 
meta-cognitive strategies. Although meta-cognition is important to all stages of the framework, meta- 
cognitive strategies are particularly important in the organization and rectification processes. Brown 
(1987) introduced a framework of meta-cognition that has two major parts, knowledge of cognition and 
regulation of cognition. Components within knowledge of cognition include knowledge and awareness of 
one’s own thinking (including capacities, limitations, difficulties), knowledge of when and where to use 
acquired strategies (including knowledge of task and situations for which goal-specific strategies work), as 
well as knowledge of plans and goals. Regulation of cognition, on the other hand, includes planning 
(setting goals, activating relevant resources, selecting strategies), evaluation (determining one’s level of 
understanding), and monitoring (checking progress and selecting repair strategies). Knowledge of meta- 
cognitive processes is central to success in both learning and development (Brown, 1987). 

Without appropriate support, students often have difficulty engaging in reflective thinking 
(Hmelo, 1999). Students must be able to consider and reflect upon the process of making decisions and 
solving problems as well as assess the quality of their understandings. An important feature in supplying 
students with collaborative learning environments designed to provide opportunities for students to develop 
shared meaning is its role in the strengthening of self-regulation and mental representations of information 
(Salomon, Globerson, & Guterman, 1989). 

Implications for computer-supported collaborative learning tools 

The explosion of computer- supported collaborative learning tools (CSCL) has encouraged 
educators at all levels to incorporate communication tools into their courses. Such tools provide areas for 
distributing expertise and sharing multiple perspectives, which can enable learners to collaboratively 
accomplish much beyond that which they could have achieved individually. In developing such 
communities, instructors encourage learners to articulate and reflect to better organize their knowledge and 
acknowledge gaps. 

Collaboration tools have been positively received by the educational community, which has 
recognized both the educational and profession values of such tools. Although the necessity of providing 
communicative and collaborative tools in distance education courses is vital, many educators also 
recognize the value of such tools in face-to-face learning environments. Professors at undergraduate and 
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graduate institutions, in particular, have consistently introduced discussion forums and collaborative areas 
designed to supplement regular class meetings. 

The many forms of computer-supported communication tools validate their usefulness and 
popularity. Geographically dispersed students such as those enrolled in distance education courses often 
use technologies such as email or teleconferencing (both text and video based) to begin to develop 
knowledge communities. New technologies, the advent of hypertext languages such as HTML, easy 
multimedia development, and access to database programs support collaboration and construction of shared 
knowledge. Bulletin board systems (BBS) and similar discussion forums have been incorporated as 
supplemental activities in many courses. 

Scaffolding perspective building through CSCL tools 

A dynamic use of communication tools is perspective building and evaluation. Having students 
contribute to a collective pool of data by sharing research, experiences, and ideas that provide multiple 
perspectives, resulting in cognitive flexibility, is just one example of the types of perspective building 
environments that could be built. Typically, students enrolled in face-to-face classroom settings share 
similar experiences, and while they are often encouraged to share their experiences, they are somewhat 
insular. Web-based communication tools provide students with opportunities to cross cultural, ethnic, and 
socio-economic boundaries, barriers seldom crossed in traditional classroom experiences. Students can 
evaluate these alternative perspectives and determine which aspects to consider, while at the same time, 
determining which to reject. 

Other tools, such as computer-supported intentional learning environments (CSILE), give students 
“responsibility to contribute to each other’s learning (Scardamalia, et al., 1989).” In CSILE environments, 
students respond to and build upon ideas and suggestions of other students. As one person changes the 
environment, the others must readapt, continually contributing to the collective knowledge. In CSILE 
environments, each student is responsible for contributing to the collective pool of data; evaluations of 
CSILE indicate that students surpass those in traditional classrooms on measures of depth of learning and 
reflection, as well awareness of what they have learned or need to learn (Scardamalia & Berieter, 1 994). 

Still other perspective-building environments such as CaMILE and SMILE (Guzdial, Hmelo, 
Hubscher, Nagel, Newstetter, Puntembakar, Shabo, Turns, & Kolodner, 1997) encourage students to share 
and respond to design ideas. The Collaborative Notebook Project at Northwestern (CoVis) provides a 
structured environment where students can record activities and ideas, view the work of others, and 
contribute to it. Such environments are meant to provide supportive structures for conducting open-ended 
activities. 

Problems with CSCL tools 

As in the case of traditional collaborative environments that indicate that some of the ways 
students in learning groups interact have been found to enhance achievement while others apparently do not 
(King, 1989; Palinscar & Brown, 1984; Webb, 1984), studies conducted to determine how students use and 
benefit from CSCL environments have been contradictory. Developers of CSCL environments have 
discovered that students focus on completing tasks and may not learn from collaborations (Helmo, Guzdial 
& Turns, 1997). Another study describes barriers to knowledge-building, in particular that students 
emphasized completion of the task; students tended to be product-oriented rather than knowledge-oriented 
(Kupperman, Wallace, & Bos, 1997). Additional studies have indicated that students often do not use 
CSCL tools as planned; the number of students who create few messages is much larger than the number of 
students who create many messages (Fishman and Gomez, 1997), discussions are controlled by a small 
number of students (Guzdial, 1997), and few authors contribute much of anything (Guzdial, 1997). The 
EduTech Institute at Georgia Tech suggests that in order to provide effective CSCL environments, students 
need to have something to talk about that must be made explicit. They introduce the idea of anchored 
collaboration, an effort to embed the discussion into ideas that are relevant and thought provoking (Hmelo, 
et al., 1997). Although CSCL tools provide powerful opportunities for evaluating multiple perspectives, 
they are not likely to be realized independently without explicit scaffolding. 

Implications for design 

Students' inadequate use of such collaborative environments prompts us to question the 
effectiveness of providing these new tools without scaffolds. In our current work with CSCL tools, we are 
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examining the role of various scaffolds of interaction strategies including: (1) areas for students to support 
their initial opinions, (2) tasks that reflect the range of possibility, (3) access to scaffolds for processes 
including comparison and reflection, and (4) access to scaffolds that aid in integration and revision. Our 
initial findings indicate that CSCL environments should include an initial opinions assignment about a 
specific topic. Initially, certain aspects of sharing, organizing, and reflecting should be scaffolded, 
including, but not limited to, the number of people who participate in the community, a description of what 
the students should look for, directions as to how they should respond, and clear directions regarding when 
to summarize. Additionally, the environment should support opportunities for re-evaluations, possibly by 
having students manipulate data with tools (i.e., sensemaker, concept maps, summarization strategies). 
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Abstract 

A growing number of training organizations are attracted to computer-based training (CBT) 
courses to replace other modes such as instructor-led training. With a growing demand for 
computer-based training, there is a need to thoroughly evaluate computer-based instructional 
software. This study describes a formative evaluation of an introductory computer-based training 
course for a financial and purchasing system. Participants consisted of 78 employees, 
representing diverse ethnic backgrounds, ages, and job categories, at a large university in the 
Southwest. Measures included achievement data from two types of assessments: a 40-item 
selected-response test of CBT content knowledge and a 5 -item test of on-line performance skills 
using tasks that simulate the real work environment. Other data collected and analyzed included 
(a) an end-of-course participant attitude survey, (b) a computer usage and preference survey 
adapted from one by Panero, Lane, and Napier (1997), and (c) data from the CBT itself on the 
pages visited, sequence of pages visited, and time spent by type of page. 

The goal in most organizations is for employees who attend training to transfer the new 
knowledge, skills, or behaviors to their work environments (Kirkpatrick, 1998; Willard, 1992). Generally, 
training departments and management personnel are interested in how well employees perform on the job. 
after receiving training. Training costs can be high in terms of non-productive time away from the job and 
for the delivery of training, especially for instructor-delivered training or where travel is involved. As an 
alternative, a growing number of training organizations are attracted to computer-based training (CBT) 
courses to replace other modes. As alluring as CBT may be, there are costs associated with development, 
delivery, and attendance hours of learners. Delivery for remote use may still require an instructor or 
facilitator in addition to network computing and other costs. Perhaps, as important or more important than 
cost, the question to be answered is do the participants learn the subject matter? Beyond that, what are their 
attitudes and perceptions related to the training? 

Formative evaluations should be considered as a continuous and integral part of the instructional 
design process with the goal of improving the instruction (Hannafm & Peck, 1 988; Popham, 1988; Weston, 
McAlpine, & Bordonaro, 1 995). Alessi & Trollip (1991) suggest that it is not a question of whether to 
conduct a formative evaluation but rather a question of how much to evaluate. Moreover, it is important to 
evaluate the training not just from the perspective of users’ attitudes, but also from the perspective of users' 
achievement (Alessi & Trollip, 1991; Kirkpatrick, 1998; Reiser & Dick, 1990; Sankar, 1988; Savenye, 
1992). 

While some organizations view formative evaluations as an integral aspect of instructional design 
with benefits in terms of cost effectiveness and efficiency, many organizations do not perceive of formative 
evaluations as a necessary part of the instructional development process (Tessmer, 1993). Formative 
evaluations can be both labor-intensive and time-consuming. An alternative often overlooked is using the 
computer-based instruction itself to perform many of the data collection tasks (Alessi & Trollip, 1991). In 
some cases, the cost of conducting the formative evaluation is considered too expensive and time 
consuming for the expected benefits of a full, formative evaluation. Sometimes, only a cursory review of 
the computer-based instruction (CBI) is undertaken (Alessi & Trollip, 1991). Often, the development team 
struggles to meet deadlines and must move on to the next project in a long queue. When time and priorities 
do not allow changes to be made to the CBT, the incentive for evaluation diminishes. The thought is why 
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evaluate if it cannot be changed at this time. Even when evaluations are conducted or when data are 
collected with the intent to conduct an evaluation at a later time, there may a paucity of achievement data. 

A recent study of corporate training programs (American Society for Training and Development, 1996) 
indicates that only 4.3 percent of those surveyed measured results, while 89 percent indicated that they used 
participant attitude surveys. Without achievement data, the evaluation can hardly be considered complete, 
especially when the final objective is for the user to perform well on the job (Willard, 1 992). 

One of the concerns in a training setting is testing or tracking performance of employees (Savenye, 
1992). Although testing and tracking performance seem to be formidable obstacles, the benefits of 
ascertaining what the student learned and areas where additional learning needs to take place far outweigh 
concerns. This is especially important in situations where a particular course is a prerequisite to other 
courses, which will build upon this knowledge. The emphasis should be on how assessing performance aids 
the instructional decision-making process and ultimately the learner, rather than on obtaining a score 
(Alessi & Trollip, 1991). For scores to be used and interpreted in a valid manner, the instruments used to 
measure attitudes and achievement need to be reliable (Thorndike, 1997). Objectives, instruction, and 
assessment need to be in alignment following the principles of good instructional design. 

Both quantitative and qualitative data are readily available from a variety of sources and should be 
used in the analysis (Alessi & Trollip, 1991; Reeves, 1993). Quantitative data on achievement, usage, 
attitudes, and prior computer experience are just some of the data which can be made available using the 
computer. Qualitative data gleaned through the comments sections of attitude surveys and through 
observations provide other sources for developing the total data picture. Path data showing the pages that 
were visited and the time spent on those pages not only supply quantitative results, but they also provide 
valuable insight into how the person interacted with the software. Analysis of time spent on particular 
activities, such as practice and instruction, can show different patterns of usage. Some participants may 
spend more time on instruction and review than practice, while others may spend less time on instruction 
and more on practice. These data may be used to assist in interpreting differences in overall achievement by 
groups of participants and to provide feedback to individuals. 

It is important to develop a plan for conducting the evaluation (Flagg, 1 990; Hannafm & Peck, 
1988; Tessmer, 1993). While time and expense are important factors, these aspects alone should not drive 
the evaluation plan. Some organizations conduct only a minimal evaluation (Popham, 1988) in which the 
main objective is to search for and correct errors or technical deficiencies (Shuell & Schueckler, 1989). 
Other organizations conduct expert reviews, one-to-one evaluations, small group evaluations, and/or pilot 
studies or field tests (Alessi & Trollip, 1991; Flagg, 1990; Hannafm & Peck, 1988; Popham, 1988; 

Tessmer, 1993). Evaluations maybe more formal in some organizations than others. 

The approach used for the formative evaluation described in this paper employs many of the facets 
previously cited. These include (a) considering the formative evaluation as an inherent and integral part of 
the instructional design and development of the program, (b) using several types of reviews, including 
expert reviews, one-to-one, and pilot-tryouts, (c) collecting quantitative and qualitative data, (d) collecting 
data via the computer, (e) measuring attitudes as well as achievement on CBT content knowledge of 
concepts, terms, and procedures, and hands-on performance tasks that simulate work tasks, and (f) 
employing commonly recommended screen-design principles to enhance motivation and provide a solid 
basis for learning the course content. 

Description of the Introduction to Advantage Training Program 

The CBT course used in this study. Introduction to Advantage, is the first in a series of courses for 
users of the Advantage on-line purchasing and financial system at a large research university in the 
Southwest. Introduction to Advantage is an eight- hour course offered in either one full-day or two half-day 
sessions to faculty, staff, or students who work full-time or part-time at any of the university’s campuses or 
other locations. In order to receive the required security access to use the Advantage financial system, 
participants need to complete Introduction to Advantage and other courses based on the level of access 
required for their particular work-related situations. Financial system users generally use the on-line 
Advantage system to order goods and services from on-campus and off-campus suppliers, to look up 
information related to the status of orders, and to monitor and maintain budget information on various types 
of accounts. 
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In the setting for this evaluation, instruction for employees (in the on-line purchasing and financial 
system) is a two-fold task. First, participants learn concepts, terminology, and procedures based on how the 
system is applied at the university, and then they learn the computer skills necessary to perform functions 
in the real on-line environment once they return to the workplace. In other words, attendees are expected to 
transfer knowledge and skills to the job once they complete the training. The initial course. Introduction to 
Advantage^ which is a prerequisite to all other courses, is designed as a self-paced CBT and hands-on 
experience using a training version of the real financial system. 

Instruction is delivered in an instructor-facilitated setting specifically designed and maintained for 
this type of training. The CBT, developed using Asymetrix Toolbook Instructor II, consists of a Quick Tour 
(which provides instruction for the essential learner-control elements used in navigation and special 
features) six content modules with instruction, review, practice and feedback, a final assessment, and a 
glossary (see Figure 1). The six content modules include (a) Overview, (b) Acquiring Goods, (c) Account 
Codes, (d) Suspense File, (e) Correcting Errors, and (f) Vendors. Figure 2 is an example of an instructional 
page from the Acquiring Goods module. 



Independent Contractor 



An independent contractor is an 
individual or organization who 
perfornns a service to ASU for a 
fee paid to the individual or 
organization. 

Within guidelines a PV may be 
used for this type of payment. 



Fisure L An example of a typical Glossary Page. 

In addition to the concept and terminology instruction for the module, modules 4-6 contain a 
series of screens that simulate specific on-line processes. Participants follow the guided instructions to learn 
the procedures for these processes prior to completing practice workshops using a training version of the 
real on-line financial system. Figure 3 is an example of a screen from the simulation of retrieving a 
document from the Suspense File. 

After the instructor welcomes the participants and introduces them to the training environment and 
materials, each participant works through the CBT at his/her own pace. Each learner receives a packet of 
materials including a quick-reference guide for use in the training and the work environment, identification 
numbers and passwords needed to access the on-line training system, and hands-on workshop exercises that 
are personalized with the individual's name and accounting information that is used by the participant's 
department. After completing all CBT modules and on-line practice workshop exercises, each learner 
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Figure 2. Example of an Instruction Page from the CBT Acquiring Goods Module. 




Figure 3. Example of a Simulation Paee from the CBT Suspense File Module, 
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completes the CBT Content Knowledge assessment and the On-Line Performance Skills (OLPS) 
assessment. The computer generates a detailed report of the learner’s results on the CBT Content 
Knowledge with scores by module. After reviewing and grading the learner's completed OLPS tasks on- 
line, the instructor then meets with the learner individually to review his or her results on the CBT Content 
Knowledge and the On-Line Performance Skills assessments. 

A similar course had been conducted successfully using CBT for almost 3 years on the financial 
system that was replaced on July 1 , 1 999. Because the newer system version employed a Graphical User 
interface (GUI), other new features and options designed to solve the Y2K deficiencies, the introductory 
course needed to be completely redeveloped. Design and planning documents were completed during the 
summer of 1998 and the CBT modules and survey instruments were developed between August and 
December 1998. Training commenced in January, 1999 and the formative evaluation for this study was 
conducted from January through March, 1999. 

Evaluation Methods 

The evaluation process was ongoing throughout the development, implementation, and subsequent 
delivery of the CBT, Several expert reviews were conducted by the development team, which consisted of 
subject-matter experts, authoring-software programmers, instructional designers, and trainers. Revisions 
and corrections were made as a result of these reviews. After the expert reviews, one-to-one evaluations 
were conducted with three employees who provide helpline support to financial system users. The helpline 
staff identified areas that needed clarification or amplification, based on their knowledge of typical 
difficulties that end users experience after completing training. These suggestions were incorporated 
wherever feasible. 

A pilot-tryout was then conducted in the normal training environment with two groups of eleven 
individuals each from a pool of employees awaiting this training. Although the development team had 
conducted some testing during installation in the training environment, a few unexpected technical 
problems were encountered during the pilot-tryouts. For example, there was a hardware situation that 
resulted in an inability to capture posttest scores for some participants in the first pilot-tryout group and 
some workstations had audio problems at one point in the software. These issues were resolved as they 
occurred or between pilot-tryout groups one and two. 



Participants 

Participants for this evaluation consisted of 8 groups totaling 78 users of the financial system of a 
large university in the Southwest. Participants attended an introductory computer-based training course for 
the financial and purchasing system from January through March 31, 1999. All training was held in the 
same training room with sessions facilitated by the same experienced instructor, who was also a member of 
the instructional development team. 

Participants in the evaluation groups consisted of 1 3 males and 65 females with diverse ethnic 
representation, although the majority of the participants were Caucasian women. Ages of the participants 
ranged from 20 to 69 with a mean age of 39. They were employed in various positions from material and 
distribution clerks to teaching faculty. The largest grouping was 46 percent in the job category of 
secretaries, stenographers, typists and clerical supervisory jobs, followed by finance-related or 
management-related specialists with almost 13 percent. Educational levels ranged from high school 
graduates to those with advanced degrees. 



Data Sources and Collection Methods 

This formative evaluation was planned as a comprehensive assessment of a newly-developed 
computer-based training course. The evaluation accounts for not only students' attitudes towards the 
instruction and the instructor, but also their achievement on CBT Content Knowledge and on On-Line 
Performance Skills tests. This information was used to improve the course and to provide individual 
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feedback on test results. The latter was especially beneficial in determining whether students had obtained 
sufficient skills to continue with the next course. 

Quantitative data were collected on-line for (a) achievement scores from a 40-item selected- 
response CBT Content Knowledge assessment, (b) On-Line Performance Skills (OLPS) assessment, (c) a 
post-instruction attitude survey, (d) a pre-instruction Computer Usage and Preference Survey (CUPS), and 
(e) path and time data. Qualitative data were obtained from comments on the post-instruction attitude 
survey. Anecdotal information that was garnered from observations by the researcher and the instructor 
will be woven into other major areas as it applies to the topic. 

Achievement Data 

Two types of achievement data were collected, using separate instruments: (a) a CBT Content 
Knowledge Assessment that measured knowledge of concepts, terminology, and procedures, and (b) an 
On-Line Performance Skills (OLPS) Assessment used to measure performance in using the on-line 
financial system training database. 

CBT Content Knowledge. 

Achievement data were gathered using a 40-item, 40-point selected-response assessment 
administered by the computer at the end of the instruction. The CBT Content Knowledge assessment 
consisted of five sections covering topics from all the modules in the CBT. Each section was self-contained 
with directions at the top of each page. Items measured knowledge of content areas from each module. 
Figure 4 is an example of an item from the Vendors Module of the CBT Content Knowledge test. 

The computer program was coded to compare learners' responses to an answer key used for 
grading and scoring. All answers were printed out and stored in a database by individual for later analysis. 
Using personalized printouts and a copy of the test items, the instructor was able to evaluate individual 
performance, to clarify any misconceptions, and to recommend areas for further study or review for each 
participant. 



Final Test Section 2 



Click the box for the most 
appropriate vendor code. 



Continue 



3. The Music Department interviewed Professor Jazz whom they 
did not hire and want to reimburse him for his travel 
expenses. 



□ MEM1 

□ A52 1645895 

□ 270449632 

□ 411000809 P 

□ INTV1 



Fisure 4. Examvle of a pass from the CBT Content Knowled£e Assessment. 
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On-Line Performance Skills Data. 

The On-Line Performance Skills (OLPS) assessment consisted of five hands-on tasks similar to 
those in each section of the on-line simulation instruction in Modules 4 through 6 of the CBT. The 
simulations and the five assessment items were designed to parallel real tasks encountered in the work 
environment. The tasks included logging onto the computer system with the correct computer application 
identification numbers and passwords, locating and retrieving specific documents from an on-line file to be 
approved, corrected, or deleted, and looking up vendor information by vendor name or by vendor code. 
These exercises were personalized with the participant’s name, individual identification numbers and 
passwords to be used, document numbers for each document to be retrieved, and vendor name or vendor 
number for looking up vendor-related information. Figure 5 is an example of one of the tasks from the On- 
Line Performance Skills assessment. 

An analytic approach was used to score the On-Line Performance Skills (OLPS) assessment. 
Scoring rubrics were developed for each of the five tasks with each worth 20 points, for an overall score of 
100. The instructor was able to. score each exercise quickly and easily by comparing workshop problems 
with the on-line transactions entered by the learner in the training financial database. All participants had 
the same tasks, but they created uniquely identifiable transactions based on the personalized information 
printed on the workshop tasks. Points were deducted for incorrect and incomplete transactions. Scores of 80 
and above were considered passing scores. Participants who scored less than 80 on the performance skills 
test were advised to review the CBT simulation for the areas missed and then to correctly and 
independently perform another set of printed practice workshop tasks, provided by the instructor. 



Retrieve the following document from Suspense, CORRECT ALL 
THE ERRORS, and CLOSE THE DOCUMENT when you are 
finished. DO NOT APPROVE AND PROCESS this document. 



Dociiment Type 


Agency 


DociimentID 


PO 


YXl 


YX109001149 



Figure 5. Example of a Performance Skills Task. 



Post-Instruction Attitude Survey 

An end-of-instruction attitude survey, administered by the computer, consisted of 26 items 
measured on a five-point Likert like scale followed by four constructed-response items. For Items 1-25 
participants rated their level of overall agreement or disagreement with statements relating to aspects of the 
course and its applicability in the person's work environment. On Item 26, participants rated the amount of 
knowledge that was new to them. Although users in general were new to the on-line financial system, some 
had been employed at the university for varying lengths of time and may have had familiarity with some of 
the terms or concepts. Participants also answered constructed-response questions relating to the aspects of 
the course they liked the most and liked the least, along with suggestions for improvement, and any 
additional comments. This survey was completed at the end of all training. 
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Computer Usage And Preference Survey (CUPS) 



The pre-instruction Computer Usage and Preference Survey (CUPS) was adapted from a survey 
developed by Panero, Lane, and Napier (1997). Each participant at the beginning of the course completed 
this 20-item survey administered on the computer in approximately five minutes. The original intent was to 
determine any correlation between a user's reported computer usage and program outcomes such as 
achievement on the CBT Content Knowledge and On-Line Skills Performance measures. Nineteen 
statements were grouped into the four categories used by Panero et al. (1997), enthusiasm, efficiency, 
entertainment, and communication. Question 20, which dealt with participants' computer use during the 
week before they attended this course, did not appear to fit into any of these categories and was therefore 
considered "Other". Twelve of the 1 8 questions that Panero et al. reported using in their paper were 
adapted, keeping the same category groupings. Reliability on the Panero et al. instrument was reported as R 
= .82. An alpha reliability coefficient of .84 was obtained in the current study. 

Path and Time Data 

The CBT was programmed to collect data about the pages that were visited, the sequence of pages 
visited, and the time spent on each page. Pages were classified and coded by type, including informational 
pages (objectives and directions), instructional pages with content (including separate optional animation 
pages), review, practice, and glossary pages. Learners had the ability to move forward and backward or to 
restart the lesson at a particular point, such as to review. Learners could click on hotwords that were 
hyperlinked to a glossary page or view animated processes. Not all learners visited the same number of 
pages. Many pages had additional information or examples that were viewed by moving the mouse to a 
check mark graphic. 



Results will be reported by each major data source. 

Achievement 

CBT Content Knowledge, 

The overall mean score on the CBT for the 78 participants was 28 out of a possible 40 points, with 
a standard deviation of 4.58. Scores ranged from a minimum of 17 to a maximum of 37, with a median 
score of 3 1 . Alpha reliability on the 40-item CBT Content Knowledge assessment was R= .70. 

The main area of difficulty for many students was the unit on paying for goods and services, as 
indicated on each learner’s printout of CBT Content Knowledge assessment results. When the instructor 
met with each participant to review results of the assessment, the payment for goods and services was the 
most common area requiring more explanation, clarification, or review. 

On-Line Performance Skills. 

Learners performed very well on the On-Line Performance Skills assessment with an overall mean 
of 94 out of 100 and a standard deviation of 1.38. Ninety-two percent of the participants received passing 
scores (80-99%) and 73 percent of the individuals received perfect scores. 

Observations indicated that, generally, most participants were able to complete these exercises 
unaided by the instructor. For those who had difficulty completing the tasks correctly, a common error was 
processing a document that was to be placed back into the Suspense File for deletion in a later task. 
Instructors could still see if the corrections had been made and in this instance only a portion of the points 
were deducted if the document was otherwise completed correctly. Selecting the wrong vendor address for 
the last exercise task was the most common error, even though there had been practice for a similar 
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scenario. The error generally occurred because the participant failed to use the suffix portion of the vendor 
code when there was a single organization with multiple locations. 

Attitude 

End-of-course attitude surveys were administered by the computer after all instruction, 
assessment, and instructor feedback to participant. Many participants took 20-30 minutes to complete the 
survey. Seventy-five percent of the participants wrote comments in the constructed-response section. Based 
on comments made to the researcher and the instructor at the end of the class, participants appeared to 
enjoy completing the survey. Responses to the end-of-course survey are summarized in Table 1, A five- 
point scale ranging from 5, SA (strongly agree), to 1, SD (strongly disagree), was used. Alpha reliability 
was .86 for the 78 evaluations completed for eight different groups over a 3-month period. 

The overall mean for items 1 through 25 was 4,4 with a standard deviation of ,31, When Items 13 
and 14 (which measured options usage rather than liking) were removed, the mean was the same (4,4) with 
a standard deviation of .34. This favorable rating indicated strong positive attitudes about the course, 
overall. Additional statements that indicate positive attitudes toward the course include Item 3, "I would 
recommend this training to others who need it," with a mean of 4,77 and Item 2, "I think the overall 
training was worth my time," with a mean of 4.73. Further evidence of participants' perceptions of the value 
of the training was found in the Additional Comments section (Item 30), in which 18 percent of the 
participants specifically cited the excellence of the training program. In the Comments Section the highest- 
rated item was a positive comment: "I like everything" given by 1 0 respondents. This was consistent with 
Item 27 (the parts liked least) in which the second-highest ranking was, "I liked everything," which one 
might interpret as "I did not dislike anything". 

From written comments on the Post-Instruction Attitude Survey and verbal comments made to the 
instructor and researcher, it was apparent that learners perceived the role of the instructor to be important in 
two ways: (a) to assist them when they had difficulties and (b) to provide feedback at the end of the 
instruction. Participants' perceptions of the importance of having an instructor (facilitator) was noted in 
three different parts of the evaluation. The second highest-rated item was Item 4, "I believe it was important 
to have an instructor for this class," with a mean of 4,79 and a SD of .44. 

In other areas of the survey the importance of having an instructor present, as well as comments 
related to the perceived excellence of instructors and design staff, was noted. In the responses to Item 27 
about the most valuable parts of the course, participants ranked the instructor as the second most important 
factor with 29 percent of the participants citing this. In the Comments portion of the survey (Item 30), the 
instructors' importance and helpfulness and the excellence of the training staff (including designers) 
received mention 53 times out of 99 comments provided on surveys in which 59 participants had written 
comments. The perceived importance of having an instructor was not limited to a particular group of 
participants and was neither associated with their achievement scores or the amount of assistance that they 
received during the course. All but one individual, who rated this item as neutral, rated the importance of 
having and instructor as agree or strongly agree 

The least-positively-rated item, with a mean of 3,1 1 and a SD of ,78, was Item 21, "The sound 
bites and music clips at the beginning of the CBT increased my interest in the course," The next lowest 
item, with a mean of 3.13 and a SD of 1.07, was Item 10, "If the materials and programs were available to 
me in my office, I would have been able to complete this course on my own". Many learners ranked these 
two items as neutral making it difficult to discern any meaningful patterns. 

Computer Preference And Usage Survey (CUPS) 

Responses to this survey indicated very high overall computer usage by participants, with 96 
percent of the participants reporting frequent or very frequent use of computers within the previous week. 
The same percentage (96) reported that they frequently or very frequently used the computer to do work 
that would otherwise take longer. Responses to the Computer Preference and Usage Survey are 
summarized in Table 2. Item 2 on the CUPS was the highest-scored item with 99 percent of the participants 
reporting that they use a computer to create professional-looking work. 

In general, participants did not report using the computer for non-work-related activities, such as, 
playing games. Ninety-one percent of the participants reported using the computer infrequently or very 
infrequently to procrastinate from doing work. Seventy-three percent reported using the computer 
infrequently or very infrequently to shop for products, such as, via the Internet, or for downloading and 
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installing software. The low usage for such activities as purchasing products or downloading software may 
be due to the numbers of participants with clerical type positions that may not permit these types of 
activities, either because of budgetary or technical controls at this institution. 

Path and Time Data 

A large quantity of data was generated by the CBT for each participant relative to the pages that 
they visited, the sequence in which they visited the pages, and the time spent per page. Descriptive statistics 
and correlations were obtained for many of these factors. The means for times take into account a wide 
range of times for various components. Several participants spent considerably more time to complete the 
overall program including the CBT portions of the program, which resulted in raising the overall average 
times for all participants. Similarly, some learners spent more time and visited more pages in specific parts 
of the instruction, especially the glossary, which effectively increased the average number of pages visited. 

Path Data. 

The average number of pages visited per individual (not counting the final assessment) was 528 
with a SD of 71. This indicates that many of the learners looked at some pages more than once. The total 
number of pages that could have been visited for these categories, if the participant visited each page once, 
was 480 pages: 129 glossary pages and 361 instructional pages. The mean number of glossary pages visited 
was 74 with a SD of 36. The mean number of instructional pages visited was 455 with a SD of 46. 

Time. 

Time was calculated both for overall time and for time spent on the CBT itself The computer 
program captured time for the CBT portion by the time spent on each page with each page coded to 
indicate the module, unit, and page type, such as, instruction, review, or practice. 

Overall Time. Overall time included all aspects of the course, such as the on-line performance, CBT 
portions, assessments, individual reviews with the instructor, and completion of surveys. The mean overall 
time for the course was 400.55 minutes or 6 hours and 40 minutes. Time per learner varied from a 
minimum time of approximately 4 hours and 30 minutes to a maximum time of 10 hours and 45 minutes. 
Thirty-eight percent of the participants completed the course in less than six hours and nine percent of the 
learners required more than eight hours. In reviewing completion time patterns, it was noted that the fastest 
learners needed just a little over four hours to complete the entire course, requiring them to return for an 
hour or less, either after a lunch break or on the next day. Those who needed more than eight hours also had 
to return on another day, usually for a period of two to four hours. 

CBT Time . The mean total time for completing the CBT portion only of the Introduction to Advantage 
course, including the posttest assessment, was 211 minutes or 3 hours and 31 minutes. Time varied from a 
minimum time of approximately 2 hours and 2 minutes to a maximum time of 7 hours and 7 minutes. On 
average, learners spent almost 3 hours in the knowledge and concepts parts of the CBT (including 
instruction, glossary, animation, review, and practice), 1 3 minutes on the posttest, and 29 minutes total for 
all simulation pages. 

Discussion 

Relative to strengths of the CBT, there are several significant findings: (a) learners performed 
adequately on the CBT Content Knowledge assessment with a mean score of 29 out of a possible 40, (b) 
learners performed very well on the On-Line Performance Skills test with a mean of 94%, (c) learners liked 
and valued the CBT as indicated with an overall mean score of 4.4 out of a possible score of 5 on the post- 
instruction attitude survey, and (d) learners responded very positively towards having an instructor present 
throughout the training session. 

Achievement on the CBT Content Knowledge assessment was acceptable overall. With a higher 
median (31) than the mean (29) it is obvious some of the learners did quite a bit better than others. In fact, a 
small number of learners with very low scores, such as 17 out of 40, account for the lower mean score. 
Another important point is that the CBT Content Knowledge posttest had six questions with poor item 
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discrimination. When these items are revised the test may yield higher scores. Learners may have acquired 
more knowledge than actually was indicated by the results on this posttest. 

Performance on the On-Line Performance Skill (OLPS) assessment is important because it is the 
terminal objective for the course and it is a representative sample of the tasks learners will perform either 
on the job and or in the next training course. Since most learners attend additional training courses that 
require hands-on practice or they perform similar tasks in their jobs, it is essential that they attain a 
satisfactory level of proficiency on the OLPS assessment. The development team set the passing score on 
the OLPS assessment in this study at 80 percent. 

Based on the mean score of 94 percent, and with 93 percent of the participants receiving scores of 
80 percent or higher, it is clear that learners performed very well on the On-Line Performance Skill 
assessment. One may, then, conclude that the simulation portion of the CBT and the hands-on practice 
workshops are very effective. The effectiveness of this combined training approach of CBT simulation with 
guided steps, followed by independent practice of tasks that closely mimic those encountered in the 
workplace setting, appears to result from the use of sound instructional design principles. 

Very few learners, 7%, did not pass the On-Line Performance Skills test. After a consultation with 
the instructor, the learners who did not receive passing scores were asked to review the CBT sections and 
complete any hands-on tasks for which their assessments indicated deficiencies. It was observed by the 
instructor and through time data collected by the computer that those who did not obtain passing scores on 
the OLPS generally took longer to complete the instruction and also earned low scores on the CBT Content 
Knowledge Skills assessment. This seems to indicate that these learners may have had other difficulties that 
need to be analyzed not only in relation to the achievement posttest, but also relative to characteristics of 
why some learners perform well and other learners do not perform well in computer-based training. 

Learners indicated that they liked and valued the course. This is an important finding because 
there are two major reasons why it is imperative that learners both learn the content and skills and be 
motivated to use the system. 1 ) Many of the clerical and administrative positions at the university are 
dependent on employees knowing how to use the financial system to acquire goods and services or manage 
the budgets of various areas and operations of the university. 2) Introduction to Advantage is a prerequisite 
to other required financial training courses. 

The intended purpose of administering the Computer Usage and Preference Survey (CUPS) was to 
begin to determine if there were any characteristics or patterns of computer use that could aid in predicting 
the successful completion of an introductory CBT with hands-on computer components without the 
presence of an instructor or facilitator. For example, could this training be offered remotely and yet 
maintain the same learning outcomes and satisfaction among participants? While the CUPS instrument had 
good reliability and yielded some information on the frequency of use by the respondents, it did not provide 
results that can be used in a meaningful way to predict the success of completing an introductory CBT for 
working adults who already rely heavily on the computer. The high importance attached to having a 
facilitator for the course is significant to any discussion relative to conducting this CBT class without a 
facilitator, which was the main reason for administering the CUPS. 

In the area of opportunities for improvement there were three findings. 1) The CBT Content 
Knowledge posttest can be improved. This was clarified by conducting an item analysis that indicated some 
problems, generally in the area of item discrimination, with approximately 15 percent of the items. 2) Most 
students performed well or very well, overall, while a small percentage performed in an unsatisfactory 
manner in the regular course time. 3) Learners generally had more difficulty with the content related to 
paying for goods and services. Since not everyone needs to complete the unit on paying for goods and 
services as a prerequisite to their next course, this unit could be moved to another course that is better 
suited for this content. For those students who took about five hours to complete the course, reducing the 
amount of course time would mean that they could probably complete the course in one 4-hour session. 
Additionally, the content should be easier for most participants to learn. 

Conducting extended formative evaluation is of benefit to organizations, especially when the data 
are easy to collect and yield information that can lead to improved training for the organization. The task 
may have been simpler at this university because the all members of the development team including the 
trainers report to' a single departmental unit. The same group is responsible for registering and scheduling 
participants, for maintaining and scheduling the training facility, for managing the telephone helpline, as 
well as for developing and delivering all financial systems training. With this array of responsibilities, the 
unit has a vested interest in helping employees use the financial system successfully. This interest and 
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desire for successful training outcomes led to the group's agreement about the need to conduct a more 
robust formative evaluation than they had in the past for other software. 

Working as a collaborative team makes revisions and adjustments easier to accommodate and it is 
beneficial in terms of consistency of approach, understanding of the resources, participants, and software 
involved. The team approach also allows continuous monitoring of the training or extended evaluations 
with data continuously captured for analysis and review. On the other hand, because the training 
development and delivery staff is small and individuals perform multiple roles, development times are 
often longer than desired. The time required for tasks such as building and maintaining a database of 
learners' results is time that cannot be spent on course development tasks. 

Analysis of the instruments used in the formative evaluation was an important aspect in trying to 
improve training. Item analysis helped identify particular areas of the posttest that may have posed 
challenges for one group of learners versus another, and this is a form of validity issue that can be 
addressed by rewriting items (Thorndike, 1997). Written and oral comments by participants, observations 
of the researcher and instructor, and a thorough analysis of the data collected by the computer were very 
helpful in pinpointing areas where changes should be made. Addressing these items by revising the CBT 
content and the CBT Content Knowledge assessment should improve the Introduction to Advantage CBT. 
Along with the course revisions, an additional formative evaluation is planned to determine the effects of 
the improvements and to ascertain if any further changes are needed. 

Designers, training managers, employees, and their supervisors need to understand and 
accommodate individual differences by providing flexibility in the scheduling of training in order to meet 
participants' and the organization's needs. Future research should be directed toward ascertaining which 
factors contribute to successful completion of CBT for some participants while others are unsuccessful. 
More specifically, are these differences due to factors in the CBT itself, are they learner characteristics or 
preferences, or are they a combination of complex interactions of several factors. Additional research needs 
to be conducted to determine why many participants prefer having an instructor or facilitator present and 
what implication this has for conducting remote training that may be more congruent with trends for 
internet and intranet delivery of instructional and educational courses. 
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Table L Introduction to Advantaze Attitude Survey Responses 







Mean 


SD 


1. 


I learned a lot from taking this course. 


4.67 


.53 


2. 


I think the overall training was worth my time. 


4.73 


.45 


3. 


I would recommend this training to others who need it. 


4.77 


.42 


4. 


I feel that it was important to have an instructor for this class. 


4.79 


.44 


5. 


I believe that this training will help me in my job duties. 


4.62 


.63 


6. 


I anticipate using Advantage in my job. 


4.50 


.75 


7. 


I am confident in my ability to use Advantage. 


4.08 


.83 


8. 


I liked setting my own pace during the computer-based training. 


4.72 


.53 


9. 


I feel that there was sufficient time for me to complete this course. 


4.62 


.69 


10. 


If the materials and programs were available to me in my office, I would have 
been able to complete this course on my own. 


3.13 


1.07 


11. 


If the program and materials were available to me in my office, I would use the 
information as a review or refresher on how to use Advantage. 


4.50 


.70 


12. 


I understood the directions (throughout the whole CBT program) on how to use 
the computer-based training. 


4.37 


.74 


13. 


I almost always clicked on the hotwords that led to more information. 


4.69 


.57 


14. 


I almost always used the check mark box that led to more information. 


4.81 


.49 


15. 


I feel that the computer animations (with arrows) helped me to learn the topic 
being presented. 


4.40 


.63 


16. 


I feel that the review at the end of each CBT section was helpful in learning the 
topic. 


4.64 


.48 


17. 


I feel that receiving feedback in the CBT practice sections helped me to learn 
the topic being presented. 


4.62 


.49 


18. 


The CBT final test helped me to understand what topics I needed to review. 


4.55 


.68 


19. 


The photos and other illustrations used throughout the CBT helped me to learn 
the topic being presented. 


4.17 


.76 


20. 


The photos and other illustrations used throughout the CBT helped motivate me 
to learn the topic being presented. 


3.88 


.88 


21. 


The sound bites and music clips at the beginning of the CBT increased my 
interest in the course. 


3.11 


.78 


22., 


I understood the directions for what to do in each of the Advantage on-line 
workshop exercises (printed on colored paper). 


4.41 


.83 


23. 


I feel that the Advantage on-line workshop exercises helped me to integrate the 
concepts and procedures I learned in the CBT with the procedures required in 
using Advantage. 


4.61 


.59 


24. 


What I learned and practiced in the on-line procedures sections (simulations of 
Advantage procedures) helped me to complete the on-line Advantage hands-on 
workshop exercises. 


4.63 


.61 


25. 


I feel that the Introduction to Advantage booklet helped me to complete the on- 


4.04 


.86 



line Advantage hands-on workshop exercises. 
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Table 2. Computer Usa^e and Preference Survey 



1 = never 2 = once in a while 3 = sometimes 4 = 
frequently 5 = very frequently 


1 


2 


3 


4 


5 


Mean 


SD 


1. 


I use a computer to save time on work that would 
take me longer otherwise. 


0 


1 


2 


17 


58 


4.69 


.59 


2. 


I use a computer to create professional-looking 
work. 


0 


0 


1 


16 


61 


4.77 


.45 


3. 


I play games on a computer. 


25 


25 


20 


5 


3 


2.18 


1.08 


4, 


I use a computer to fill free time. 


19 


20 


29 


6 


4 


2.44 


1.10 


5. 


I lose track of time while using a computer. 


10 


17 


16 


27 


8 


3.08 


1.22 


6. 


I use a computer to procrastinate from doing work. 


57 


14 


6 


1 


0 


1.37 


.69 


7, 


I use a computer to communicate with people I see 
regularly in person. 


11 


13 


21 


18 


15 


3.17 


1.31 


8. 


I use a computer to keep in touch with friends and 
family who are far away. 


6 


12 


16 


17 


27 


3.60 


1.31 


9. 


I spend time downloading and/or installing software. 


31 


26 


12 


7 


2 


2.01 


1.08 


10. 


I use computer reference manuals and on-line help 
features when I need to know how to do something 
different or new on the computer. 


3 


15 


22 


26 


12 


3.37 


1.08 


11. 


I call a computer support hot line when I need to 
know how to do something different or new on the 
computer. 


9 


38 


22 


7 


2 


2.42 


.90 


12. 


I shop for products using the computer (on the 
Internet, etc). 


37 


20 


11 


7 


2 


1.92 


1.11 


13. 


I spend time learning about the computer by 
attending classes. 


5 


25 


26 


19 


3 


2.87 


.99 


14. 


I spend time learning about the computer by self- 
study and by experimenting. 


0 


6 


11 


42 


19 


3.95 


.84 


15. 


I shop for computer hardware or software by going 
to stores or looking at catalogs. 


24 


23 


13 


16 


2 


2.35 


1.19 


16. 


I listen to music on my computer. 


22 


12 


19 


16 


9 


2.72 


1.38 


17. 


I use the computer to find out the latest news 
happenings or check stock prices. 


23 


20 


12 


15 


8 


2.55 


1.36 


18. 


I forward interesting items that I find on the Internet 
to my friends and family. 


18 


19 


15 


18 


8 


2.73 


1.33 


19. 


I use my computer to create or send greeting cards to 
my friends and family. 


17 


23 


19 


13 


6 


2.59 


1.22 


20. 


During the past week, I used my computer. 


0 


1 


0 


7 


69 


4.81 


.70 



n = 78 (except for questions 1 2 and 20 where n = 77) 
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In traditional print journals scholarly dialogue occurs over a protracted timeframe. It often takes 
six months to a year for commentary to be published. The goal of this school psychology e-joumal project 
was to facilitate scholarly dialogue among researchers, practitioners, and graduate students. The electronic 
journal we developed will allow professionals in school psychology to engage in professional dialogues in 
a much shorter timeframe. 

Jack Cummings, a professor in school psychology at Indiana University, procured agreements 
with editors of six major print journals in the field of school psychology. The editors were willing to select 
“best articles” - one or two annually from each journal - to be published electronically. Professor 
Cummings, who served as the client for this project, asked us to conduct research and to develop a Web site 
which would facilitate dialogue among practitioners and scholars around these “best articles” which were 
expected to be provocative and possibly controversial. 



Methodology 

Since the major objective of the team was to design a useful and usable web site, the approach we 
chose in creating this e-joumal was user-centered design (Boling & Frick, 2000; Corry, Frick & Hansen, 
1997; Dieli, 1989; Dumas & Reddish, 1993; Gould & Lewis, 1985; Nielsen, 1994, 1999; Rubin, 1994; 
Shackel, 1991). We focused on the users throughout the project. We looked at user feedback as the driving 
force of the design and development process. This is reflected by the fact that every major step during this 
project was accompanied by user input of some kind and revisions were made accordingly. 

Needs Analysis 

To define the purpose and the content of the web site, the team conducted needs analysis 
interviews with both the client and some potential users. Boling and Frick (2000) define needs analysis as 
the following: 

”A needs analysis is an investigation that seeks the evidence to answer two important questions 
about your web site: 

• What do you and the other people in your organization (the stakeholders) want from the people 
who will use your web site? 

• What will the users of your web site want and need from it?” 

According to Boling and Frick (2000), needs analysis is important in that it defines “the major 
purpose of your site, and a clearly defined purpose will help you decide what the content of your site 
should be." 

As a first step to needs analysis, we identified the client needs. Initially, the client gave us a 
presentation describing what he had in mind. To further clarify his intention, we conducted several face-to- 
face meetings and corresponded e-mails with him. Through these needs analysis, we identified the 
following: 
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• The goal of this web site is to facilitate the communication between practitioners and researchers. 

• The primary audience of this web site includes practitioners, researchers, and graduate students in 
school psychology. The secondary audience of this web site includes parents and teachers. 

• The client wants this e-joumal to include the following basic components: 

■ A list of lead articles in four interest areas; the articles will be selected by editors 
of source journals. The source journals for this e-joumal include Journal of 
Educational and Psychological Consultation, Journal of Psychoeducational 
Assessment, School Psychology International, School Psychology Quarterly, and 
School Psychology Review. As sources journals can provide articles in varying 
interest areas, this e-joumal will carry articles in four areas; that is. Consultation, 
Intervention, Prevention/Health Promotion, and Assessment. The lead articles should 
be readable on-line. 

■ A public discussion forum: Users can post comments about the lead articles 
sharing their opinion about them. 

■ Refereed comments: Among the comments posted in the public discussion 
forum, some comments will be selected by editors of this e-joumal for their quality. 

These selected comments will be published in a separate page for easy recognition. 

After conducting need analysis with the client, we attained initial understanding about the purpose 
and basic format of the web site. Then, we conducted phone interviews with seven potential users to hear 
their opinion about the site and their needs as a user. Three of the interviewees were practitioners, three 
were graduate students, and one was a faculty member in school psychology. Each interview took around 
20 minutes. We introduced the project and the web site briefly. Then we asked six questions regarding 
whether they were going to use the site, what features they would need in general, what their opinions were 
about a public dialogue, what features they would need to have an effective and efficient public dialogue, 
etc. Through the interviews, we identified the following: 

• There is an apparent need for a web site like this. People want to get articles and communicate 
with other people. They think a web site like this will be a good way of doing so. 

• People are naive about e-joumals. They do not have much of an idea about what an e-joumal 
would look like and would provide. Those who have ideas relate an e-joumal to a print journal or 
article databases such as ERIC or Psyinfo. 

• Three different kinds of activities are expected to be carried out through this web site: Article 
placeholder, communication place, and paper publication place. 

• Practitioners would like to get practical articles and students would like to get research articles. 
However, all of them are interested in getting good practices and innovation articles. 

• When thinking about communication routes, people like to have interest group communication 
and personal e-mail communication as well as open discussion. 

• Features: Good search (by topic specifically), download articles, chat room, easy-to-use in 
general, quick in getting needed articles, etc. 

• There needs to be a moderator for good quality articles and discussion. 

Task Analysis 

To decide on the functions of the web site, the team conducted a task analysis where we 
considered results of the needs analysis and practical limitations we had. To get a realistically feasible list 
of functions, we needed to negotiate among client needs, user needs and our limitations. The needs 
analysis interviews with the users gave the team clearer understanding about the web site and some issues 
that we needed to discuss with the client. It seemed that the users welcomed this web site and liked the 
basic components that the client had in mind, e.g., four main interest areas in School Psychology, a public 
discussion forum, refereed comments. In addition to these components, users indicated the following needs. 
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• Database of articles. 

• Interest group discussion. 

• Personal e-mail dialogue as well as public dialogue. 

• Editor mediation or membership control to get good quality articles and comments. 

• Quick and easy-to-use web site. 

The users’ additional needs posed us the following discussion issues: 

• What is the main function of the web site? We know this site is going to be a communication 
place. However, users want to have a database of articles. Do we provide such a database? 

• How do we categorize content? By topic? By journal? 

• How do we assure quality of the content? People want to get good quality articles and 
communication as well as free discussion. 

• How do we form interest groups and manage membership for the groups. 

• What communication options do we give? Forum, e-mail, and others? 

• We need to think about the maintenance of this site. 

We discussed above issues within the team first and with the client next. Then, we again met with 
the client to decide on the functions of the web site which will become the basis of our paper prototype of 
the web site. The followings summarize what we have decided in the meeting: 

• The main focus of this web site is to facilitate public dialogue rather than providing articles. 

• The articles will be categorized by four interest areas. 

• The team will use the Web Bulletin Board System (Web BBS) to develop communication 
features within the project timeline. 

• E-mail function will be provided to facilitate private dialogue. However, the main focus of this 
web site will be given to public dialogue, not to private dialogue. 

• Editors of this e-joumal will moderate the quality of comments somewhat by deleting very bad 
comments. There won’t be any membership control. 

• Easy maintenance tool will be provided for the client. 

• Interest group dialogue will not be developed through this project considering our limitations. 

Creating the Paper Prototype 

Based on the results from the need analysis and the task analysis, the team created a paper 
prototype. The paper prototype helped to “get a quick idea of how the web site was going to be organized 
and where major elements should be placed.” (Boling & Frick, 2000) Our focus was on the overall 
structure of the site. 

The methods we used to create this paper prototype included brainstorming and expert 
consultation. We initially brainstormed on the structure considering needs analysis and task analysis 
results. Then, we consulted with a usability expert and the client. The expert was knowledgeable about the 
process of paper prototyping and the client was knowledgeable about the content of the site. 

To create the paper prototype, we conducted several important meetings within the team as well as 
with the expert and the client, where some of the major decisions on the layout of the web site were made. 
In those meetings, we sketched the site on papers trying to place contents and features where they most 
suitably belong. We requested the client to provide us with any content needed to create a paper prototype. 

Conducting Usability Tests with the Paper Prototype 

To identify major problems with the overall structure of the site, we conducted usability tests 
using the paper prototype. According to Boling and Frick (2000), usability test participants should be 
authentic users whose key characteristics match a profile of the potential users of the site. Therefore, we 
asked the client to provide us five potential users because he had contacts with some authentic users. One 
of them was a university faculty member in school psychology, three were graduate students in school 
psychology, and the last was a high school psychologist. 
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To prepare for the usability test, we wrote a test introduction script highlighting some important 
issues such as the purpose of the test and confidentiality about the data and participant profiles. We also 
created an observation form which we will use to record our observations while the participants are 
conducting the test. Lastly but most importantly, we wrote ten usability tasks. The tasks were authentic 
tasks; that is, they were the tasks that users of this web site would perform typically using the site. For 
example, users were asked to find a specific research article, to make a response in the forum to a comment 
made by another user, to subscribe to an e-mail discussion list, etc. 

In the usability test we asked the participants to complete the usability tasks and observed whether 
they completed the task or not, how many seconds they spent to perform each task, what paths they took to 
complete the tasks, what comments they made etc. One of the team members administered the test while 
other team members observed. However, every team member could jump in and ask some follow-up 
questions whenever necessary. We rotated our roles within the team so every team member could have the 
opportunity to do the administration as well as the observation. Most tests were done in a computer lab, but 
we went to the participant's place when he or she preferred it to coming to our lab himself or herself 

The usability test revealed the following problems with the paper prototype 

• On a few pages, people did not know where they were and what they could do in the pages. 

• Some terms were confusing because they were used inconsistently. Others did not convey their 
meaning well. For example, people did not know what “refereed submissions” meant. 

• The distinctions among the three types of commentaries: “general comments”, “case studies” 
and “empirical research” were not clear to the users. 

With above usability results, we made following revisions to the paper prototype: 

• We wrote specific instructions on the pages. For example, before the article titles, we added 
instructions that described what users could do by clicking on the titles. 

• We changed certain terms so that they were consistent with other terms used in the prototype and 
clearly understandable to the users. For example, we changed “refereed submissions” to 
“refereed commentaries”. 

• We wrote descriptions about each type of the commentaries so users understand the type easily 
and decide where they want to go next. 

Creating the Computer Prototype 

The next thing we did was to develop a computer prototype of this e-joumal. A computer 
prototype is the translation of a paper prototype into marked up HTML files that can be viewed with Web 
browsers. As the first step, we determined the standard elements of the pages. They include header, footer, 
and left-side bar menu. Header is the top part of the pages. It orients users about their location within the 
site and on the World Wide Web. The header consists of a logo image, title of the site, and links to search, 
help, and about us pages. Footer is the bottom part of the pages. It consists of last update information, 

URL of the document, comments, and copyright. For easy access to the top-level pages, we placed several 
links on a left-hand side bar. The side bar includes links to home page, four interest areas pages, and 
sponsors page. By using it, users can quickly get to the top-level pages from any levels of the page 
hierarchy. When users get to a specific page, the link to that page becomes inactive and the background 
underneath the link changes indicating that the page is being displayed currently. 

Secondly, we created a visual identity of the site. Using the standard elements described in above 
paragraph, we created pages that have a similar look. The similar look across the pages created a visual 
identity specific to our site. It will help users to know that they are still looking at our site no matter how 
many links they followed from page to page. 
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Figure I. The home page of the computer prototype. 
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Lastly, we developed template pages for the whole site considering the standard elements and the 
visual identity. Template pages are the generic pages for each level of the site hierarchy. 

Conducting Usability Tests with the Computer Prototype 

By testing at this point of the development, we were able to find problems with the site in a more 
realistic environment than the paper prototype usability test. In addition, by creating and testing templates 
pages that is, a prototype rather than the whole site, we were able to improve on the problems without 
investing too much. 

Before conducting usability tests with authentic users, we conducted pilot tests with expert users. 
A design expert and a subject matter expert tested the template pages and made suggestions for 
improvement. According to the test results, the following major revisions were made: 

• For the top-level pages, "prevention/ health promotion" link in the side bar was renamed to 
"prevention". We reduced size of the title of the site in the header part. In addition, the 
sentence “Best articles and commentaries on assessment" was added as one of the main 
components of the header. 

• For the article home page, the title of each article was added and its font color was changed to 
dark violet. 

• For the forum home page, horizontal navigation menus ( i.e.. Home > Assessment > Public 
Forum) were used instead of the left-hand side bar menus. 

With the revised prototype, we conducted usability tests with seven authentic users. The profiles 
of the participants were similar to those of paper prototype test. The main profiles we considered when 
selecting participants were occupation, Web knowledge, and content knowledge. 
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We used the same usability tasks with those we used for the paper prototype test. Participants 
were asked to post a discussion topic and a response, send an e-mail, find a message, etc. These tasks were 
authentic tasks for the site and focused on the interaction between the users and the site. 

The tests were conducted using the computer in our computer lab or in the participant’s home or 
office where they usually connected to the Internet. At least two team members participated in the usability 
testing. One member administered the test and the other member observed the test. After the usability test, 
we asked some follow-up questions. 

The usability test resulted in the following findings: 

• The low success rate was due to the poor design of the Web Bulletin Board System mostly. 
The Web BBS was not flexible enough for us to customize for our site. 

• People tended to use the links at the main body of the screen rather than the links on the side 
bar. 

• Success rate did not depend on the degree of web knowledge or content knowledge. 
However, it depended on the carefulness of the users. More careful readers tended to 
complete more tasks. 

Suggestions that the participants made during the tests included: 

• Need a better word for "Refereed." 

• Need an index of articles by authors. 

• Add a category of "Policy and Professional Issues." 

• In Home Page, provide descriptions about the four categories to facilitate the users with low 
content knowledge. 

Comments that the participants made after the tests included: 

• They will use "Read Articles" and "Refereed Commentaries." However, they are not sure 
whether they would participate in the forum. 

• They like this new idea and will use the site. 

The usability test raised several issues that we could not resolve immediately. The most serious 
usability problems occurred on the Web BBS pages. However, the Web BBS was not flexible enough for 
us to improve on the problems. Therefore, we recommended that we should find and purchase a more 
flexible Web conferencing system. Another issue was that users did not use the sidebar menus frequently. 
However, we could not conclude that the side bar is unnecessary with this limited usability testing results. 

Although there were a few unresolved problems, we revised the prototype as much as we could. 
The final version of our computer prototype can be seen at http://www.indiana.edu/~eioumal . 



Site Management Issues 

The client needed a cost-effective and efficient way to manage the site. He also wanted this site to 
be updated by a person who does not have much web development skills. In response to his needs, we 
developed a web file editor that creates files and directories automatically. We also created a job-aid for 
using the editor. With a password-protected Web file editor and a step-by-step job aid, a novice developer 
can manage the site. 

Summary 

In this research and development project, we followed principles from user-centered design. We 
began with a needs analysis, both with our client and the primary target audience. Based on these results, 
we did rapid prototyping. First, we developed a paper prototype and conducted usability tests with 
authentic users. We observed problems they experienced when attempting to conduct typical tasks with the 
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paper prototype. Next, we developed a computer prototype, which incorporated design changes warranted 
from the paper prototype usability test results. We conducted further usability tests with the computer 
prototype and revised it according to problems we observed. One major problem area we could not fix was 
the interface to the Web bulletin board software that we incorporated into the computer prototype. We 
observed significant usability problems during our tests with this shareware. At the time of this writing, the 
client is considering purchase of a commercial Web conferencing tool with better usability and which can 
be adapted to fit into the current design framework. When that is accomplished, then the School 
Psychology E- Journal will go into the final production stage and be implemented on the Web, 
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Abstract 

This manuscript reports the results of a qualitative study focusing on the ways in which 
technological tools are implemented specifically in mathematics education in a Title I school 
Specifically, the intent was to understand the perspectives and actions of the school's mathematics 
specialist and the multi-graded 2nd-3rd-grade classroom teacher as they attempted to deliver 
instruction with technology for both English as a Second Language (ESL) and non-ESL students. 
Results showed that a critical factor in access to mathematics education and technology for 
immersion students in a multi-graded 2-3 classroom in a Title I K- 5 school setting is language. 



Introduction 

In recent years, education technology has been infused into our nation’s schools with the hope that 
it will improve the academic performance of our students {Education Week, October 1, 1998). Wenglinsky, 
(1998), presents findings from a national study (1996 National Assessment of Educational Progress in 
mathematics) of the relationship between different uses of educational technology and various educational 
outcomes. Reading literature on educational technology and media reports on this matter, a number of 
questions came to mind; How do teachers define technology? What does technology mean to the people in 
the school setting? What does the use of technology in mathematics education look like? In November, 
1998, we embarked on a qualitative study to understand the use of technology in the mathematics education 
process in an Arizona Title I K-5 school. Interviews with the school’s mathematics specialist, classroom 
teachers, observation in the classroom and Higher Order Thinking Skills (HOTS) laboratory and analysis of 
documents were used to gain an understanding of what is happening at the site. 
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With the publication of the National Educational Technology Standards for Students (ISTE, 

2000), and the revision of the National Council of Teachers of Mathematics’ (2000) Principles and 
Standards for School Mathematics coinciding in time, new emphasis has been placed on the importance of 
implementing powerful learning technologies into the mathematics classroom. The technology exists to 
allow children, even very young children, to explore complex and advanced mathematical topics, keep 
track of their learning histories, and represent their thinking in varied and complementary forms (NCTM, 
2000). Implementation of such powerful tools, contrary to the perspective of popular culture, places firm 
emphasis on the role of the teacher in facilitating and channeling students’ activity to develop in a 
mathematically sound and sophisticated manner. 

Calculators are ubiquitous. Computers and hand-held devices are following suit. In addition, new 
models of technology delivery are being instituted that make use of projection systems, local area networks 
and features of the world-wide-web, such that all students in a mathematics class can interact with 
technologies significantly, everyday — even when the ratio of computers to students is low (Middleton, 
Flores, & Knaupp, 1 997). 

The technology exists. Access is available. Exemplary practices have been identified. So, what is 
happening? At a gross level, we know that technologies are implemented differentially across grades and 
socioeconomic status of schools. However, within a classroom, we have few analytic models of how 
teachers come to develop practices that make use of technological tools, and how those practices impact the 
learning of students. What constitutes “typical” implementation of computational media in a public 
mathematics classroom? How are technologies used as tools for deep mathematical understanding? This 
study was conducted to provide an analytic frame for understanding this question. 

Methods of the Study 

The study is situated within an interpret vist perspective (Erickson, 1986). Data was collected over 
a period of six months. The first author of this research report conducted the participant observations and 
the second author assisted the former with the data record keeping, methods of interviewing, analysis, and 
construction of the research report. Sources consisted of multiple observations of the classroom, computer 
lab and English as a Second Language (pull-out) classroom; interviews with the mathematics specialist, the 
classroom teacher, and the ESL and reading specialists; and analysis of curriculum and assessment 
documents. 

Participant Observations 

Over 30 hours (1 hour per day, twice per week, for 1 5 weeks) were spent observing the 
interactions in the classroom, the computer lab, and the ESL classroom. Under the assumption that the 
primary purpose of computational media in the mathematics classroom is to provide multiple ways of 
communicating ideas, exploring claims, and articulating arguments (Pibum & Middleton, 1 998), the aim 
for each observation was to capture the classroom discourse as it pertained to these issues. The features of 
the discourse noted during observations were: 1) Technological tools used; 2) Mathematical 
Representations developed; 3) Arguments made (by students); and 4) Teacher Facilitation. To capture the 
relationship between arguments made and teacher facilitation, the teacher was audio taped continuously to 
obtain a verbatim record of her directions and interactions with students. These tapes were transcribed and 
integrated with the observation notes. 

Teacher Interviews 

Semi-structured teacher interviews were geared towards gaining an understanding of each 
teacher’s philosophy and practice related to technology for mathematics instruction. Open ended 
statements, questions such as, “Please share with me infonnation about your classroom. What are the 
school and district goals for mathematics? Please share definitions of technology. How do you envision the 
use of such technology in the classroom?” and so on were posed. Prompts were used to get access to the 
teachers’ perceptions. The classroom teacher, the mathematics specialist, the ESL teacher, and the reading 
teacher were interviewed at intervals of two-three weeks over the course of the study. Interviews lasted 
approximately thirty minutes each and were audio taped and transcribed. 
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Document Collection 

Since technologies function as tools among other tools and materials that facilitate instruction, to 
understand this context, we collected artifacts from the district, school, teacher, and student levels. The 
materials we collected included curriculum materials such as the grade level thematic units, the district’s 
curriculum with district articulated learner outcomes for mathematics, model math lessons, assessments, 
and related scoring guides. District-compiled assessment results (the 1998 Stanford achievement test), state 
academic standards. Fauna’s assessment plan and school goals were collected. Student-generated work, 
sheets of paper used by students to work out mathematics problems, sample worksheets that were assigned 
as homework, and class handouts were collected as well. In addition, user manuals and teacher guides for 
software used by these students were made available to the researchers, as well as printouts of screen 
dumps and student work on computers. 

Analysis of this data began with the most local context: Student work, handouts and curriculum 
materials to provide the substrate within which student and teacher discourse developed and around which 
the discourse revolved. School and district-level materials were used subsequently to provide an 
understanding of the barriers to technological implementation and the dilemmas faced by the teachers in 
supporting student learning through technology. 

Analysis of Data 

As the observations and interviews progressed, initial hypotheses about the role of technology in 
the plans and practices of teachers were noted. As regularities among the ways in which technologies were 
used, and the ways in which teachers and students discussed mathematics using technology-afforded 
environments and representations allowed us to induce firm hypotheses concerning why these patterns 
existed in this particular context. 

A number of propositional assertions regarding the induced hypotheses were developed after a 
thorough reading of the entire data record. From this reading and the initial questions with which the study 
began, two assertions that were supported empirically survived the subsequent search for disco nfirming 
evidence. In the results of this manuscript, these assertions are presented with evidentiary samples that 
embody the data corpus. These samples are in the form of quotations from teachers and students, 
descriptions of teaching methods, classroom interactions from the observations, excerpts from teacher 
interviews, and descriptions of classroom structure. 

The Setting 

Fauna offers an English as a Second Language (ESL) program that pulls Limited English 
Proficient (LEP) students (mainly monolingual Spanish speakers and recent immigrants) from their regular 
classes for English language instruction. The monolingual Spanish speaking children, are referred to as 
“immersion children,” referring to the belief that the more the ESL children are exposed to English, the 
more English they will learn. Over 80% of Fauna’s 800 students receive free or reduced price lunch. 
Fauna’s student population is 30% Limited English Proficiency (LEP), 62% Hispanic, 25% White, 5.2% 
Native American, 4.3% African American, and 3.5% Asian/Pacific Islander. The school buildings are 
approximately 40 years old. After World War II, land grants were given to the Hispanic community around 
the school and some of the families have lived in the area for generations. However, over the last ten years, 
new immigrants from Mexico have arrived in the area. The school has a considerable transient population, 
with families moving in and out of the area. Fauna reported a mobility rate of 59% in 1997 and absentee 
rate of 6.3% in 1998. One block away from the school is an abundance of motels, trailer parks, and pay-by- 
the-week housing, which contributes to the mobile nature of the student population. Right next door to 
Fauna Elementary is a state-of-the-art community center run by the city, with amenities that, among other 
things include basketball courts, a ftill gymnasium, and computer facilities. The school uses the center for 
Physical Education activities and some of the students attend after-school programs at the center. 

Each classroom has a Macintosh G3 computer acquired in Fall, 1998, and/or one or more older 
Macintosh SE’s. Fauna has a math specialist, Mrs. Trimble, who is paid from US Department of Education 
Title I ftinds and is in charge of a “Higher Order Thinking Skills” (HOTS) laboratory with 20 Macintosh 
G3’s. Approximately eighty staff members, including administrative, certified, classified, and support staff 
are at Fauna. Four ESL teachers, four ESL aides, and four reading specialists also form part of the school 
staff. A year-round calendar at Fauna divides the school year into four sessions each separated by a three- 
week recess and the school year culminates with a six-week summer recess. 
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The Classroom 

Mrs. Stravinsky’s classroom is the only multi-year 2nd-3^‘^ grade classroom in Fauna Elementary. 

In late April 1999, at the completion of the study, the class had twenty-five students; eleven 2"^^ graders, 
and fourteen 3*^^^ graders with thirteen boys and twelve girls. This class was formed by selecting students 
whose families had lived in the neighborhood for a generation or two and deemed fairly stable. Mrs. 
Stravinsky started the school year with twenty-one students. Gabriel arrived from Mexico in late November 
1998 and Jesus, also from Mexico, arrived in late February 1999. However, over the course of the school 
year, three or four students left the school and some new students arrived as well. The classroom’s ethnicity 
is one Native American, one African American, one Vietnamese, eight White, and fourteen Hispanic 
students. Twenty of the twenty-five receive free or reduced price lunch and twelve are classified as ESL 
students. This excerpt from an interview with Mrs. Stravinsky, November 28, 1 998, illustrates the teacher’s 
perception of her students” parents’ ability to help them in their learning at home: 

“For many of them, the parents don’t speak English, then they can’t help the child in 
English. They may be able to help in mathematics, it crosses all lines, but they can’t help 
in certain areas. But, very limited. Many of the parents that I’ve come across are illiterate 
or they speak another language.” 

This statement reflects the expressed belief of Mrs. Stravinsky and also Mrs. Trimble (supported 
throughout this study) that “Mathematics is a language of its own” and is less impacted by the use of 
language than other subject areas. 



Results of the Study 

In the context of Fauna Elementary the following empirical assertions were made as a result of this 

study: 



• Assertion 1 . The instructional practices of teachers show a broad local definition of 
technology that coordinates representational tools and presentation systems under the 
umbrella term, “Teaching Tools.” 

• Assertion 2. For the immersion students, language is critical in providing access to 
mathematics learning and technology and thus to academic success. 



Assertion 1. The instructional practices of teachers show a broad local definition of technology that 
coordinates representational tools and presentation systems under the umbrella term, “Teaching 
Tools.” 



For the purposes of this research study, at the outset the researchers defined technology as 
computer software, Internet, graphing calculators, electronic devices, and other similar devices used as 
tools in the mathematics education process. However, the researchers took care not to take this definition 
for granted. Local meanings of technology went beyond this initial definition to include math manipulatives 
such as pattern blocks, geo-boards, fraction tiles, etc. Use of these manipulatives was observed consistently 
during the time spent in the classroom. Mrs. Trimble, the math specialist, vocalized and demonstrated her 
definition of technology. During an initial interview, when asked how she perceived technology in 
mathematics education, Mrs. Trimble said: 

“I see technology as any tool that I use to help get across the concept. Whatever concept, 
standards, performance objective I’m teaching, technology is the math manipulative, the 
computer, the overhead; the tools I use to help the students, either in delivery, practice, or 
assessments. Technology as manipulatives or blocks — pieces of things that they can 
explore on their desks — helps build the visual concepts.” 
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Field observations corroborate the local meanings expressed by the teachers. A pattern emerged in 
the regular use of pov^er blocks, math-fraction tiles, pattern blocks, geo-boards, and paper & pencil. An 
excerpt from field notes of an observation occasion in Mrs, Stravinsky’s classroom illustrates this: 

February 23, 1999. 

Mrs. Trimble: What we 're going to do today is we are going to do a lesson on relative 

area and it is going to look like what we have done in geometry already 
and what we have started in fractions. We are going to use a 
manipulative called power blocks. And power blocks have like 3 
different kinds of triangles, squares, rectangles and parallelograms. 

They are relatively new. So I'm kind of piloting them. 

The power blocks consist of ten different sized triangles with each triangle numbered T1 , T2, and 
so on. There are five squares, SI through S5, five rectangles, R1 through R5 and four parallelograms, PI 
through P4 in different colors and increasing size. Mrs, Trimble held up different shapes and asked the 
students to recognize them and say out aloud the name of the shape. Mrs. Trimble gives students five 
minutes to explore the power blocks and explains to the researcher: 

Mrs. Trimble: Now, when they are exploring they are going back to whatever they 

have mastered like making patterns or sorting by shapes, something 
that they feel very comfortable with. But because these are pretty new 
and novel, they are touching them, they are playing with it. I don 't 
think they have gone through exploring this before. They 're just going 
to build after they go through this other thing. They 're going to explore 
whether you give them time or not. They'll turn you off during the 
lesson and explore, so, if not with their hands, with their eyes, while 
you are trying to teach. This is the best five minutes spent. 

Mrs. Trimble and the researcher walked around the classroom to discover that some of 

the students sorted the blocks by shape and size. 

Many such instances of classroom use of math manipulatives were observed. The teachers hoped 
that the use of the manipulatives contributed to mathematics learning experiences that were designed to 
support the formation and communication of mental conceptions among learners. The researchers propose 
that this local definition of technology embodies the notion of cognitive technologies as promoted by Pea 
(1987). Moreover, it supports the representational perspective on technology (e.g.. Kaput, 1995) in that 
such materially realizable notation systems contribute to the formation of mathematical conceptions. 

Use of software programs in the HOTS computer lab was also observed. Students from Mrs. 
Stravinsky’s classroom spent at least two class periods a week in the HOTS lab where software such as the 
Logical Journey of the Zoombinis’^'^ (Broderbund Software, Inc.) and Turbo Math Facts 3.0 (Nordic 
Software Inc.) was used. These pieces of software served a distinctly different purpose than the 
manipulative tools. Whereas the manipulatives were used as ways of revealing students thinking, sharing 
strategies, and arguing about the structure of mathematics, the computational tools were used mainly as 
drill and practice scaffolds. Students worked alone on the stand-alone programs, did not interact with each 
other or the teacher significantly, and did not discuss strategies or underlying mathematical ideas. This 
contrast is marked, in that while using manipulative tools, the classroom more resembled the kinds of 
classrooms that emphasize the development of mathematical understanding (Hiebert, Carpenter, Fennema, 
Fuson, Human, Murray, Olivier, & Weame, 1997). While using computational tools, however, the 
classroom more resembled a didactic environment that emphasizes fact production. 

These inconsistencies are not uncommon as teachers struggle to implement reformed practices 
(Koellner & Middleton, 1999). However, the fact that they were so isolated from each other leads us to 
believe that computational technologies are viewed and implemented utilizing an entirely different set of 
practices and pedagogical content knowledge than other technologies. The computational technologies 
were isolated geographically in the HOTS lab instead of integrated into the classroom, the software was 
designed utilizing a contradictory model of what is important mathematically and pedagogically than the 
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classroom environment and the grouping practices and discursive structures of the classroom and HOTS 
lab were completely different. 

If technologies are available, schools are wired, and exemplary models for implementation are 
available, it appears that they are not being implemented consistently and in a fashion that is integrated 
with classroom instruction. 

Assertion 2. For the immersion students, language is critical in providing access to mathematics 
learning and technology and thus to academic success. 

The very first day, Mrs. Stravinsky mentioned that she has students who hardly know the letters of 
the alphabet. As the researcher spent more time at the site, a picture emerged that helped the researcher see 
how language played an important role in the student-teacher interactions and, hence, a significant role in 
the student’s access to learning. 

(Excerpt from Interview with Mrs. Stravinsky, March 11,1 999) 

We have children in here that are considered immersion children, means that they speak 
another language and relatively no other language. Although Marie la, when she arrived 
here was having little command of the language and she’s already reading at level 8. In 
our equivalency, level 8 is the pre primer. I’ll just read this level 8 to you. 

Baby ’s birthday. It was baby ’s birthday. 

She got a book. She played with the paper. 

She got a pair of shoes. She played with the box. 

She got a rocking horse. She played with the ribbons. 

She got a cake. She played with the candle. 

We all sang happy birthday. Then baby went to sleep. 

Lot of repetition. She got a, she got a, she got, she played with, she played with. 

Varied reading levels of students 

The students are at different reading levels. Mrs. Stravinsky has students who cannot identify 
letters of the alphabet. Mrs. Miller, a reading specialist, works with the Limited English Proficient (LEP) 
students, some of whom are recent immigrants and monolingual Spanish speakers. Mrs. Miller works with 
them five days a week for one class period each day. She is the only reading specialist in the entire district 
who can speak both English and Spanish. Mrs. Miller works with small groups of four or five students for 
twelve-fifteen minutes each at the reading center. The students are formed into three groups, with one 
group at each of the three centers: 

1. listening center — a tape recorder is connected to a device which allows multiple headphones to be 
plugged in and an audio taped book is played for the students. 

2. a reading center which has hands-on visual aids — ^where students play word games matching 
visual aids like a toy bear, car, etc, with cards that display a letter of the alphabet and a picture of 
the visual aid. The students say B b bear. . . 

3. the reading center where Mrs. Miller sits at a U-shaped table with the students facing her and she 
reads to the students, asking them to follow along with her. 

These centers are adjacent to each other at the back of the classroom. While Mrs. Miller works 
with the LEP children, Mrs. Stravinsky works with the other half of the class on various reading related 
activities. The following vignette illustrates a typical approach to literacy: 
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It is a bright Arizona spring morning, cool and pleasant, in Mrs. 
Stravinsky’s multi-graded 2-3 classroom. Mrs. Stravinsky goes over the 
day’s schedule with her students who are sitting at their desks and is 
settling herself down at her U-shaped reading table at the front of the 
classroom. There is an announcement on the PA system “Reading 
Specialists, Mrs. Elder and Mrs. Glenn are out sick today. Students 
who work with them should stay in their classrooms and not go to the 
ESL classroom.” Mrs. Stravinsky says, “None of you work with Mrs. 
Elder. Those of you that work with Mrs. Glenn, stay here.” She reaches 
out to the bookshelf next to her and pulls out a sheaf of papers, copies 
of a lesson on choosing the right word. She looks at the clock and the 
bell rings. This is the first class period of the day. 

In walks Mrs. Miller from the back door pushing a cart. The cart has a 
number of white plastic boxes neatly arranged, some are filled with 
small, thin books, others have plastic bags with toy like visual aids and 
word cards. She places five of these plastic bags at the round table next 
to the listening center. She settles down at the U-shaped table at the 
back of the classroom. Children walk in from the back door, the side 
door connecting to the adjoining classroom and from the main door. 
Enrico, Gabriel, Maria, and Mariela from Mrs. Stravinsky’s class join 
the others from other classes and gather around Mrs. Miller. 

Mrs. Miller has twelve children around her. Speaking in a low voice so 
as to not disturb Mrs. Stravinsky, Mrs. Miller, giving instructions in 
Spanish and English, forms three small groups of four students each. 
Enrico, Gabriel, Maria, and Mariela, the four monolingual Spanish 
speakers from Mrs. Stravinsky’s room are at her center — the reading 
table. The others are at the listening center and the reading center. 

Mrs. Miller holds up a chart, which has the letters of the alphabet in 
upper and lower case, and related color pictures. For example; A, a, and 
a picture of an ant. She holds the chart facing the children and says with 
a smile: “Let us go over the letters of the alphabet. A a ant.” The 
children point to the drawing of the ant and repeat after her “A a ant.“ 
Mrs. Miller goes through the letters of the alphabet B b bear, C c car, D 
d dog, E e elephant, F f fish and so on. As the students join in chorus, 
the volume from this table chanting the letters of the alphabet can be 
heard across the classroom. Enrico put his head down on the table. Mrs. 
Miller touches his arm gently, directs him in Spanish to sit up and pay 
attention; “Enrico, escucha.” Gabriel, Maria, and Mariela are looking at 
Mrs. Miller and following along. Enrico yawns. Done with the alphabet 
chart, she puts the chart back on the wall behind her. 

Mrs. Miller reaches up to her cart and carefully pulls out a little book. 
She leans forward with her elbows on the table and holds up the book 
“Building with Blocks” with the cover facing the students, points to a 
red colored block pictured on the cover page and says, “What color is 
this?” Mariela says, “Red.” Maria stares with a blank look. Enrico is 
yawning again. Mrs. Miller says, ”R-r-red.” Mrs. Miller says to the 
researcher, “Mariela could not speak English when she arrived early 
this year. Now, she’s understanding what she’s reading. She can read a 
lot more than she understands. She knows all her letter sounds, and is 
just rolling along, everyday. The boys have just arrived from Mexico, it 
will take a while” and sighs. 
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Each child at the table has a copy of the book. The book is very 
colorful and has large pictures of children playing with blocks, and the 
letters are in a large font. Mrs, Miller reads the book twice, asking the 
students to repeat after her each time, while pointing to the colored 
block on the page. Again Mrs, Miller provides these instructions in 
Spanish first and then in English, Mrs, Miller says, “Lean conmigo,” 
“Read with me.” She starts the reading; “A green block. A red block, A 
yellow block, A blue block. A black block, A white block, A 
spaceship,” Enrico and Maria yawn, Enrico puts his head down again. 
Mrs. Miller tells him in Spanish to pay attention. “Escucha, Enrico,” 
Mrs. Miller looking at the students, trying to keep eye contact with 
them says, “Open to page 3. Find a word,” She holds up three fingers to 
indicate page 3. Mariela points to Red, Mrs, Miller smiles in 
acknowledgement and says “r-r-red. Point on A-Z chart,” Maria and 
Gabriel point at “n.” Mariela points at “r.” Enrico doesn’t respond. Mrs, 
Miller repeats slowly “r-e-d,” Enrico seems very tired; he looks as if he 
has not had enough sleep the previous night. He puts his head down on 
the table again. This time Mrs. Miller doesn’t say anything. 

Mrs. Miller collects the “Building with Blocks” books and gives them 
another book, “At School.” Enrico is nudged by Mariela who is sitting 
next to him and he sits up. Mrs. Miller says, “Diga conmigo. Say with 
me.” Again, Mrs, Miller reads the book twice, with the students reading 
with her and pointing to the words as they say it aloud, Mrs, Miller lays 
the book flat on the table and uses her index finger to trace the words as 
she reads. Mrs. Miller reads, “I like reading. I like writing. I like 
singing. I like playing, I like painting, I like my teacher. My teacher 
likes me.” Mrs. Miller pauses and smiles, makes eye contact with the 
students and says, “Look on page 8. Find ‘my,’ the letters ‘m’ ‘y’.” 
Mariela and Maria find “my,” Gabriel glances shyly around his 
neighbors and follows, Enrico yawns, looks at Mariela and points at 
“my.” Mrs. Miller collects “At School” from them and gives each 
student two books to read at home, one that they have already covered 
in school and the other that is new to them. 

Mrs. Stravinsky has the main door open, one of her groups is in the 
corridor and the other two groups are in the classroom. The group 
outside is reading chapter 6 of a book with a partner: “It’s like This, 
Cat” by Emily Neville, 1963, Harper Trophy: New York, NY, ISBN: 0- 
06-440073-5. This book is a winner of the Newberry Medal. Mrs, 
Stravinsky has Joshua, Jewel, Cathy, and Peter at her reading table. She 
says, “Look at what you need to do this morning. Everybody, read with 
your eye while I read out aloud, please. This is part two. Remember, 
normally this is a test. We are reviewing it. So when you take your test 
in the spring you are ready. You need to do Questions 5 through 16, 
Question 5. Choose the word that finishes each sentence correctly. Fill 
in the space on the answer sheet with the best answer. Now look at the 

sample. It says Jason enjoys all outdoor . Is it: a.) sport 

b.) sported c.) sporting d.) sports,” Looking at Joshua, Mrs. Stravinsky 
asks, “What do you think it is?” Joshua replies, “d, sports.” 

Mrs. Miller’s group is singing a song and Mrs. Miller’s gentle voice 
with the children in chorus wafts over to where Mrs. Stravinsky is 
sitting: 
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“B b Did you ever see a brown bear, a brown bear, a brown bear, eat 
blue berries and bugs” 

Pointing to the letter A, Mrs. Miller asks, “What letter is this?” 

Gabriel says in a low voice, “I.” Mariela and Maria say, “A.” Enrico 
just stares at Mrs. Miller. Mrs. Miller smiles and says, ”A a The ant ate 
the apple, A a The ant ate the apple, High-ho the demy-o, The ants ate 
the apple.” 

It is evident that the monolingual Spanish speaking students, referred to as immersion students, are 
at beginning kindergarten reading level in English, while others are able to read chapter books. Fauna has 
the unique challenge of orchestrating a classroom with students who are native English speakers, students 
who speak no English at all but speak fluently in their primary language, and a large number of students 
according to the teachers, mostly recent arrivals, who speak no English at all and have a poor background 
in their primary language as well. The policy is that instruction is provided exclusively in English. Teachers 
are unable to provide immersion students the explanation of an important concept in the student’s primary 
language. Krashen, 1996, suggests that first language use in class occasionally is useful rather than teachers 
resorting to the frustrating use of pantomime and gestures; however, cautions about situations where the 
input is hard to understand, that concurrent translation is necessary. He espouses the idea of sheltered input. 

Language as access to learning 

As the observation occasions progressed it was noticed that the immersion children seemed 
disinterested during instruction, which was primarily in English. Mrs. Stravinsky, the classroom teacher, 
and Mrs. Trimble, the math specialist did not speak Spanish, even though they made attempts at 
communicating with their limited knowledge of the language. During mathematics instruction, teachers 
typically used a math manipulative, allowed students to explore on their own with the manipulative, and 
gave instructions on how to use them. Instructions are in English, and students with little or no knowledge 
of English are easily distracted and engage in activities of their own that are not related to learning. 

An analysis of the district’s learner outcomes for mathematics revealed that 2"^ grade students 
were required to “solve word problems using the appropriate operations.” This seemingly simple learner 
outcome is certainly a challenge to immersion students who are beginning to learn the letters of the English 
alphabet. The class was asked to solve math word problems as outlined in the “problem of the day” that 
was done daily. Either the math problem of the day would be announced on the television at the beginning 
of the day for each grade level or Mrs. Stravinsky would put up a problem on the board. An analysis of the 
district’s Description of Content and Process Clusters Grades 3-5 for the Stanford Achievement Test, 
showed that one of the goals is for students to be able to compute in context. Under the Mathematics 
Procedures for Grade 3, “Computation in Context: Solve every day problems requiring addition, 
subtraction, and multiplication of whole numbers, and addition and subtraction of decimals and money” 
was listed. Here is an example of the problem of the day that allowed for practicing “Computation in 
Context.” 

(Excerpt from Observation Occasion in Mrs. Stravinsky’s classroom, March 1, 1999) 

Problem of the day.* My daughter Bridget borrowed $275 from me. She paid me back 
$127. Now she gave me a check for $411. How much do I keep? How much will she get? 

Most of the non-LEP students were attempting to solve this problem. However, it was 
clear that the immersion children, who had arrived at Fauna at various times of the school 
year, were unable to follow the instructions provided in English. Most of the immersion 
students were engaged in scribbling on the paper, while a few attempted to copy down 
the numbers that their neighbors were putting down on their sheet of paper. 

“Language” began to emerge as the key to access mathematics learning. In the HOTS lab, students 
sat at a computer each and worked on software that allowed them to practice basic math facts (Turbo Math 
Facts 3.0). However, the instructions are again verbal (in English) from the software and/or from Mrs. 
Trimble and the prompts on the screen are in English as well. The immersion students either simply sat at 
the computer clicking the mouse at random or occasionally, when seated next to another ESL student, 
ended up talking quietly among themselves. In the classroom environment, Mrs. Stravinsky and Mrs. 
Trimble attempted to communicate with the immersion students in whatever little Spanish they knew and 
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attempted to have other bilingual students translate their instructions from English to Spanish for the 
benefit of these students. However, during the thirty hours of observation, fellow bilingual students seldom 
translated for the monolingual speakers. The immersion students demonstrated behaviors that indicated 
their loss of interest in the mathematics learning opportunities that were offered: 

(Excerpt from Observation Occasion in Mrs. Stravinsky’s classroom, February 18, 1999) 

Mrs. Trimble: (to the researcher, explaining the objective for the lesson) 

My objective today is writing fractions. Read, write and speak the fraction, and 
also to know the different values, which one is smaller and which one is bigger. 

Mrs. Trimble: (to the class) 

/ have brought in my fraction bars and we are going to do some fraction bar 
building. And Tm going to see if you can write a fraction, read a fraction, and 
build a fraction. So we are actually going to do something that you ’re not 
supposed to do until and 5^^ grade. We’re actually going to add mixed 
fractions, which you are not supposed to know how to do just yet. So don ’t tell 
anybody. Okay? (In a conspirator’s tone.) 

Class: (Enthusiastically responded.) Yeah! 

Mrs. Trimble gave students five minutes to explore the fraction bars/tiles. The tiles were 
rectangular in shape, with 1 black tile representing one “whole,” 2 orange tiles 
representing a 1/2 each, 3 green l/3rds, four purple l/4ths, five blue l/5ths, six red l/6ths, 
eight brown l/8ths, ten yellow 1/lOths, and twelve white l/12ths. We walked around the 
classroom and were observing what the students were doing during exploration. We saw 
that Maria had her head down on the desk. 

Mrs Trimble: (to the researcher, referring to Maria) 

You can see that she is not ready 
Mrs. Trimble: (to Maria) 

You are not feeling too well today Maria? 

Maria did not respond verbally. She Just lifted her head up a little. 

Mrs. Trimble: (to Maria) 

Just need to explore right now . . . 

We walked over to Juan. Maria had put her head down again. 

Mrs. Trimble: (to Juan): 

You are ready for the lesson? Huh? What are you doing? 

(Pauses.) 

Juan smiled shyly. Juan had the large fraction bar considered to be one whole and on it he 
was building a pyramid with the other fraction bars. He attempted to cover up his creation 
with his arms. 

Mrs. Trimble: (to Juan and the researcher) 

Just exploring, seeing how they fit together, very interesting. 

Mrs. Trimble: (to the class): 

Okay. What we ’re going to do today is, now, we ’re going to use these (referring 
to the fraction tiles) as tools for learning, now that you ’ve used them to explore. 

I saw some pretty cool exploring going on! 

(Pauses.) 

Okay, I want you to hold up the black piece. What does the black piece have on 
it? What numbers? 
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Students hold up the black piece in the air. Most students did, Gabriel and Jesus, the two 
monolingual Spanish speaking children seemed to follow what the other children were 
doing and catch up by holding up the black piece. Enrico and Mariela are attempting to 
build some elaborate structure with their fractions bars. 

Class: One. 

Mrs. Trimble: 

One. Guess what? This is your *‘one whole, ** This is the whole, this is the one. 
(Holding up the black piece, which is a rectangle shaped plastic piece.) 

( Pauses.) 

Okay, I want you to see if you can make the ONE with the oranges (still holding 
up the black piece, which was designated as "one whole.”) 

The students lowered their hands. 

Gabriel was yawning. He made eye contact with the researcher and smiled shyly. 

Mrs. Trimble: 

Lay the black (referring to the tile^ on your table and build it with the oranges. 
Not on top, but right below or right above it. 

Okay. One whole and you are going to build it with the oranges, exactly. 

Pauses a few seconds (20 or so). Some of the students like Tony, Joshua, Cathy had their 
fingers up in the air showing “two” by holding up two fingers. 

Mrs. Trimble: 

How many oranges did it take? Show me with your number fingers, some of you 
already did. 

Okay, two. 

Who can raise their hand and tell me the fraction that is on the orange tile? 

Sara: One out of two. 

Mrs. Trimble: One out of two or one half. 

Jane held up one hand and the other hand below it, on the wrist of the first hand with two 
fingers, indicating one half Maria had her head down on the table. Juan was busy playing 
with the tiles. 

Mrs. Trimble: 

And Jane remembered from yesterday how we said we could do fractions by 
holding up our one hand with our numerator and holding up the other below for 
our denominator. She remembered one half 

Okay. Let me show you what you are not supposed to know until about or 5^* 
grade. Are you ready? 

Class: Yeah. 

Mrs. Trimble: 

Today, you 're going to write some algorithms. That means math sentences. 
What does one half plus one half equal? 

(Pauses a few seconds.) 

Does anyone know? (Walking towards the other end of the room.) 

What do you think it equals? 

Connie: Two halves. 

Mrs. Trimble: Two halves. And what does two halves equal? 

What do you think it equals (looking at Tony). 

Tony: One whole. 

Mrs. Trimble: Oh! Lets write that. 

(Writing on the board: 1/2 + 1/2 = ) 

An algorithm just means an adding sentence with numbers. 
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(Pauses.) 

Look at this (pointing to what she just wrote on the board.j This is only for 
and grade, 

(Pauses) 

One out of two plus one out of two equals, , , 

Sarah: 2/2 

Mrs. Trimble: Or, , , (Pauses). 

Joshua: One whole 

Gabriel was playing with his tiles, looking around at others. Jesus was now opening and 
shutting the flap of his shirt pocket that had Velcro. 

Mrs. Trimble: It equals one, 

(Completes writing on the blackboard 1/2 +1/2 = 2/2 = 1) 

One! 

Okay now 1 want you to build the black whole. , , Isn 7 that kinda cool ''black 
whole? ” , , , using orange and blue. Just one or two oranges and some purples, 

Mrs, Trimble continued with the lesson, leading the students through with the building of 
the one whole with smaller fractions. She allowed time for the students to build the one 
whole on their own, providing them an opportunity to test and explore on their own. 
However, it is necessary to point out that almost none of the immersion students 
responded to the prompts from Mrs, Trimble attempting to involve them in the lesson. 

While Mrs. Trimble was leading the lesson, Mrs. Stravinsky walked around the 
classroom assisting students, or worked with one or two of the immersion students and 
their bilingual neighbor, if any, attempting to get the point across. 

Natural language is used along with mathematical notational systems — both symbolic and 
visual — during the process of mathematics instruction. Structured experiences provided by the teachers for 
the students during mathematics instruction used the academic language of English. Instructions on how to 
use the math manipulatives and the computer software also involved the exclusive use of the English 
language. Immersion students did not have any means of understanding the complex “problem of the day,” 
which was presented in written English. The minimal Spanish vocabulary of the classroom teachers proved 
insufficient in reaching the immersion students as evidenced by the student actions. The reading level of the 
2 nd_ 3 rd immersion students was at a pre-kindergarten or kindergarten level. A few of them were 
unable to identify letters of the English alphabet. Observation of immersion students’ interactions with 
mathematics technologies indicated that they had little idea of how to operate the technology, let alone 
learn with the tools. This situates the quality of access immersion students have to mathematics learning 
and technology. Particularly for the immersion students, sessions involving the use of manipulatives 
designed to explore math concepts degenerated into playing with them. Sessions involving the computers 
became random key punching and/or off-task behaviors. 

Use of computer software at a level that allows students to explore their understanding of the 
process of addition and subtraction in multiple representational systems was not in evidence. Multiple 
mathematical representations that are visual and symbolic allowing students to make the move from simple 
to more complex and abstract notions are recommended for understanding (Nickerson, 1995, Roschelle, 
1996). Most software use instances observed were at the level of practicing facts, using the Turbo Math 
Facts 3.0 software. This could be attributed to the lack of commercial educational software to enhance 
mathematical understanding. The use of software programs that purely encourage practice of basic facts 
suggests that remedial instruction exists in conflict with the use of hands-on math manipulatives that 
allowed students to explore complex abstract notions in mathematics. In spite of such use of the powerful 
learning technologies in mathematics education, both the hands-on learning opportunities and the practice 
software use remained inaccessible to immersion students. 
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Multiplicity of other factors affecting learning 

Social factors play an important role in the school life of students at Fauna Elementary. It was 
noticed that before school starts and when school lets out, there are very few parents leaving their children 
or picking them up at school, indicating that mothers and fathers are working. Most children rode the bus. 
Teachers repeatedly referred to the home conditions of the students as deprived of reading materials and 
lacking opportunities to engage in reading and/or conversation with English language speakers. 

Researchers observed students who seemed tired, and wearing unwashed clothes; students going to the 
nurse’s office for shoes or appropriate clothes to attend PE; and students with a cough/cold almost all the 
time. It is important to acknowledge the complex nature of life and the various issues that contribute to the 
phenomena. 

A report by the district psychologist showed an analysis of Fauna’s performance in the Stanford 
Achievement Test for spring 1997 and spring 1998. The mean percentile rank for grade 3 students at Fauna 
in spring 1998 indicate that they are at the 29^^ percentile in reading, 21^^ percentile in mathematics, and 
32"° percentile in language. Mrs. Stravinsky mentioned in an interview that this district psychologist told 
them that given the number of reduced and free lunches, he could predict the SAT percentile for the grade 
level. 



(Excerpt from Interview with Mrs. Stravinsky, November 23, 1 998) 

Xeah, / do believe that they (students) are at a disadvantage because of their socio- 
economic background. There's a lot of violence in many of their lives and then 
academically if you look at a variety of standards, I know on the Stanford 9 it had shown 
that our scores in comparison to the free and reduced lunch. I mean, there 's a definite 
correlation between that. And the particular psychologist that spoke to us said that, if we 
were to give him the percentage of free and reduced lunch he could predict what their 
Stanford 9's would be. And knowing the other schools in the district, because my two 
daughters attended a school within this district a different background obviously, but 
their particular school scored higher in the Stanford 9, but they had a lower percentage 
of free and reduced lunch. 

This seems like an example of the school district, as an institution, making the correlation that 
socio-economic background, as determined by the free and reduced lunch status of the students, is related 
to their academic success, as measured by the Stanford Achievement Test. During one of the observation 
occasions, as the researcher was checking in with the school secretary, it was noticed that the school 
secretary was unpacking a huge box which contained several copies of the same book; Payne, Ruby K. 
(1998) A Framework for Understanding Poverty. RFT Publishing: Baytown, TX. The school secretary 
shared that the Principal and Assistant Principal had attended a workshop where they came across this book 
and felt that it was necessary for each one of the teachers at Fauna to read it. As a result, each teacher was 
given a copy of the book and a study guide. A formal memo from the Principal accompanied the book 
asking that the teachers read it over the intersession. Thus the school administration was making an attempt 
to help teachers understand their students’ background. 

The teachers expressed that the immersion children seldom have an opportunity, at home with 
their parents or outside of the school environment, to converse in English and are seldom exposed to print- 
rich environments. Access to computer programs and software programs outside of school for a majority of 
Fauna’s student population is extremely limited. Teachers perceive that most immersion students’ parents 
are illiterate. Repeatedly in interviews teachers mentioned that most parents at Fauna were unable to help 
their children with academics. Parent participation in their children’s education as perceived by the 
classroom teacher is very minimal. 

(Excerpt from Interview with Mrs. Stravinsky, March 11,1 999) 

Oh! Well. Well, it’s interesting because right now. I’m taking two master’s classes. There 
are teachers from districts all over the valley. They speak about how the parents at their 
schools have these different library drives; they secure books for the libraries. And they 
even meet them at the bookstore, at Borders! And they’ll sit and have coffee with the 
teacher, and they’ll just discuss literature. And then I have to say, ’’well, that doesn’t 
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happen everywhere , " And so I have to share with them the reality here is that some of 
our parents have never been to a library, poor things. And we just had this occur last 
weekend. Our librarian took a family to the City Public Library, They had never been to 
the library before. First experience. And the one little girl is in this class and / had one of 
the sisters ' last year and then there *s another sister that *s in fifth grade. So, both the 
mother and the father showed up and they don V speak English and the librarian gave 
them the tour of the City Public Library, And they lost the mom, and they were running 
around looking for the mom. Well, they found her, she was in the section with the Spanish 
language books, and she had her nose in a book, she was reading. She was so excited. 

And obviously the family is literate in a language and I didn 7, I really didn 7 have any 
idea if they were or not. All five of them have their own library cards now. But, what*s 
interesting too is the fact that the librarian shared the story with me, and not the little girl 
in this class. And I was surprised because I would think that she would share that. But, 
then she is so quiet. So, she just didn 7 share the fact that the whole family went to the 
library. So, you know, I mean there *s an interest. But, true there *s such extreme poverty 
here and illiteracy here that this doesnt always occur. And this is the first time that the 
librarian *s ever done this, that had a family that came to take the field trip. 

This exceq^t illustrates that not all immersion students’ parents are illiterate. Yet, the teachers 
perceive that Fauna’s immersion students do not have a better education in their primary language. Better 
education in student’s primary language is indicative of greater subject matter knowledge. However, 
immersion students do not have caregivers who are able to help with school work and the time and 
knowledge to interact with the school; and have access to materials in their primary language and English 
at home. While the school district psychologist and the teachers seem to subscribe to the view that low SES 
is causative of low achievement, the researchers subscribe to the view that SES is not causative (Krashen, 
1996). In addition to comprehensible input in English, subject matter knowledge in the primary language, 
and literacy development in the primary language is recommended (Valdes, 1998). However, Fauna’s 
district has adopted an English as a Second Language program, where total immersion seems to be 
recommended. Fauna has a broad and progressive definition of technology that includes math 
manipulatives and software programs; yet, the primary focus is on immersing students in English. With this 
approach the use of mathematics software programs in Spanish though available, are not an option. 

In summary, the practices of “immersion” as defined by the district. Fauna school, and the 
particular behaviors of Mrs. Stravinsky and Mrs. Trimble, were found to be exclusionary in terms of 
students whose first language is not English. These practices, when coupled with technologies that require 
the use of English to interpret merely added insult to injury, as the context of one-person-one-machine 
isolated children from each other and the teachers, who could have provided some basic translation support. 
In addition, the very tasks the software programs provided students were drill-and-practice oriented, and 
provided no other support mechanisms for coming to understand arithmetic patterns. 

Discussion 

Results show two incommensurate sides to technology implementation. On the one side, teachers 
utilize powerful learning technologies (manipulatives and other modeling tools) as a natural part of their 
everyday practice. These tools are used to reveal thinking, to offload working memory to free up executive 
functions, and they are used as tools for communication of underlying, foundational mathematics. On the 
other side, computational technologies served none of these functions (what Pea, 1 987 calls the 
transcendent functions of technologies). Why? 

We suggest four hypothetical reasons for this disconnect. The first deals with software availability. 
The software available to Mrs. Trimble and Mrs. Stravinsky doesn’t do the kinds of things that the 
manipulative tools do. Despite the commercial availability of exemplary packages. Fauna school has yet to 
purchase any of these packages. Thus, even if the teachers had a perspective on computational media that it 
should facilitate the transcendent features of technologies, the tools they have at their disposal prevent any 
of these features to be taken exploited. 

Second, disconnect exists in the teachers’ knowledge of exemplary software. Whereas we have 
already established that exemplary software were not available, the teachers did not have any experiences 
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(through demos, etc.) of the kinds of computational learning tools that they might want to purchase. 
Whereas manipulative tools have received the lion's share of coverage (in teacher journals such as 
Teaching Children Mathematics, or in staff development packages), reviews of software, and especially 
research on its design and use has not been widely available in a format teachers can understand and use. 

Third, Fauna school utilizes an impoverished model of technology integration (as it pertains to 
computers) as an institution. Infusion of computers and projection systems into classrooms where they can 
be integrated into instruction (Middleton, Flores, & Knaupp, 1997) are recommened over the separate 
computer laboratory model. Fauna's district has opted for continuation of the separate computer laboratory 
model. This policy preemptively separates computational technologies from the everyday learning 
experiences of children, and thus exempts children from developing habits of mind that make best use of 
the kinds of representational and communication tools that computational media afford. To be 
appropriately integrated into the mathematics curriculum, technological tools need to be available when 
and where they are needed (NCTM, 2000). 

Fourth, school and district policies constrain teachers' abilities to deal effectively with the issues 
of language. While the teachers studied believed that mathematics is its own language, separate from a 
learner's primary tongue, their behavior suggests that mathematics teaching is, regardless of this belief, 
predicated on effective communication. In Fauna school, effective communication is heavily weighted 
towards the use of English language instructions for teaching, and verbal English representations of 
understanding for learning and classroom discourse. Because the computational tools made available to 
teachers did not facilitate the more robust kinds of communication strategies and representations suggested 
in the standards documents, Spanish-speaking students were excluded from legitimate mathematical 
discourse and instead exhibited behaviors reminiscent of learned helplessness such as checking out, 
daydreaming, or acting out (e.g., Dweck, 1986). 

Teachers at the school hold an inclusive definition of technology that exceeds the traditional view 
that technology is primarily computer-based programs. While the local definitions of technology include 
manipulatives and other tools that could function to bridge the language gap, seldom were the immersion 
students able to access these technologies due to their lack of access to the language in which important 
concepts and classroom management decisions are negotiated. Manipulatives, graphs and charts enable 
learners to explore and form initial understandings of complex, abstract notions in mathematics; yet, due to 
the fact that subject matter instruction is in English, the use of the tools by the dominant culture is 
incomprehensible to students without facility in the dominant language. Results of this study have 
implications for school policy and practice regarding the role of English in the use of technology for the 
education of recently arrived non-English speaking children. The most important implication is that the 
classroom teacher/instructional technologist must take into account the role of language when designing 
and delivering mathematics instruction and technology to such students by ensuring students understand the 
academic instructions that are used in combination with the notational systems, and that the emphasis in 
discourse be that of communication (e.g., NCTM, 1989), instead of verbal English, This may suggest the 
use of “comprehensible input” in sheltered subject matter (mathematics instruction) learning (Krashen, 
1996) as a way to facilitate technology integration for all. Providing equitable access to learning, academic 
success, and fully developing the intellectual potential of all learners is an enormous challenge (Valdes, 
1998). The complexity of layering computational technologies into instructional practices compounds these 
issues. 

Clearly, there are a host of other issues that set the context at Fauna; social issues of poverty, race, 
immigration or citizenship status; policy issues of ESL vs. Bilingual Education; funding issues with regard 
to the hire of bilingual educators, district policy and teacher/administrator perceptions of Limited English 
Proficient/immersion students and so on. Recognition of the complexity of Fauna's context, as a reasonable 
case of technology implementation, helps shed light on the historical difficulty schools have had providing 
equitable access to their second-language learners. 
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Abstract 

In this paper we discuss scholarly communication and the emergence of electronic journals. 
Scholarly electronic journals are expanding their influence. Currently, about 70 peer-reviewed 
scholarly journals in education are freely available through the World Wide Web (http://aera- 
cr.ed.asu.edu/links.html/). Wider access to scholarship, new possibilities inherent in the electronic 
medium, and a shortened publishing lag are some of the issues relating to this expansion. Also, the 
collision of the financial aspects of traditional print publishing with the ground rules of academic 
scholarship is contributing to the progress of electronic dissemination of scholarship. Concerns 
about the quality of scholarly electronic communication are misplaced, since the peer review can 
be even more rigorous with on-line journals than with traditional print publications (G. V Glass, 
1994). In addition to economic issues, there are a number of medium-specific technical features of 
electronic publication that offer a huge potential to revolutionize scholarly communication. 
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Introduction 



Scholarly publication is at the core of academic life. Changes in the nature of scholarly 
communication are certain to reverberate throughout academe. Since Gutenberg, technology has played a 
key role in shaping the nature of scholarly communication. In the 1 970s, the advent of desktop publishing 
and the use of computers changed the nature of scholarly publication. Now, telecommunications 
technology is shaping the way we view the dissemination of scholarship. One could hardly overestimate the 
importance of these new media in advancing the cause of scholarship and science; surely Stephen Hawking 
did not. On December 26, 1999, on the Larry King Weekend telecast on CNN, Stephen Hawking was 
asked. 



“[KING] What, Professor Hawking, do you consider the most important discovery of this 
millennium?” 

and Hawking replied, 

“[HAWKING] I think the invention of printing was a breakthrough for the human race. It 
meant that information and discoveries could be disseminated widely and not just on a 
one to one basis by word of mouth or handwritten manuscript. It led to an ever increasing 
rate of scientific and technological development. This has now made printing almost 
obsolete and replaced it by the Internet.” 

The role of commercial publishers as the sole purveyors of scholarly work is being threatened. 
Commercial publishers were the first to see the potential and take advantage of the microcomputer 
revolution — the advent of desktop publishing in the 1970s. Their numbers expanded remarkably and their 
profits soon followed suit. Because the capital investment required to enter this arena was minimal, the 
industry has become flooded with small-time publishers eagerly hunting for manuscripts. The scale of 
publication was transformed radically on account of low production costs; 500 copies became a “break 
even” point for scholarly books priced at as much as 25 cents/page. Quantity in number of titles came to 
replace quality. It appears to casual observers that very little peer review remains in the commercial 
publication of scholarly books. A very conservative estimate of 1 13 commercial publishers and 74 
university presses in the U.S. that currently publish books on education can be found at 
<http://coe.asu.edu/edrev/publish.htm>. It is safe to say that in the year 2000, five or ten times (per capita) 
as many scholarly books will be published as had been published at a time — 1950’s — ^when to publish a 
book represented a significant investment in resources by a large publishing house. Traditional scholarly 
publishers have exercised a great degree of control over the publication process, based on financial 
considerations first and access considerations last. The degree to which the cost of print journals has risen 
has forced university libraries to cut back on their print acquisitions and examine new technologies. 

Today, electronic mail and the World Wide Web have become essential tools of the academic. The 
relatively low cost of access to this technology in contrast with that of print communication has made 
scholarly electronic journals an affordable alternative. However, the sheer volume of information appearing 
on the Web has raised concerns about the quality of on-line resources (Dillner, 1999). We suggest that this 
concern about quality is misplaced. The emergence of electronic media has afforded scholars the 
opportunity to make available their research work to the public “free of charge,” effectively changing the 
nature of scholarly communication. Previously marginalized populations now have access to what was 
once outside their reach. The dialogue between educationists, policy makers and the public is enhanced (G. 
V Glass, 1999b). 

Technical issues of bandwidth, media type, and capabilities afforded by the new media in relation 
to the nature of electronic journals are worthy of examination. Effective use of videos and audio clips with 
text can easily alter the nature of the scholarly reports with regard to the validity of inferences in the 
research reports. The ability to make large data sets available at relatively low cost allows other scholars to 
run secondary analyses on primary data. However, this raises issues of bandwidth and the ability of 
electronic publishers to make use of appropriate technology. The affordances of this technology challenge 
scholars to prepare research reports that take full advantage of the media. 
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Electronic Scholarly Journals 



Peer-reviewed electronic scholarly journals (ejoumals) have at once become a novel, economically 
viable mode of scholarly communication and a challenge to traditional means of publishing research 
journals. The print journal has been in existence since the mid 1 7* century and, by its very nature, it has 
remained static and representative of one-way communication. Print journals are published at a certain 
moment in time, usually available as a subscription for a price, or included with membership to a scholarly 
society. Ejoumals are only a decade old, with the ability to provide information that can be timelier, and 
access can be free. Although the reader is dependent on access to a computer, rapid distribution directly to 
the desktops of subscribers ensures timely delivery in geographically remote locations (Luther, 1 997). 

The written word in print journals has allowed the dissemination of scholarship to many; however, 
this medium does not allow for exchanges between authors and readers of a kind akin to serious 
discussions. Conversations among researchers that occur in conferences at formal symposia and other 
informal forums allow for interactivity that is impossible to achieve in print media. Yet, absent the web, 
such interactions are restricted to a specific geographical location, time, and the confluence of people and 
interests. Ejoumals offer the opportunity to overcome these obstacles to include interactive exchanges that 
are not restricted by geographic distance and time. Steven Hamad of Princeton University, a leader in 
exploring new modes of scholarly communication, predicted about ejoumals that they would — . . restore 
scholarly communication to a tempo much closer to the brain’s natural potential while still retaining the 
rigor, discipline and permanence of the refereed written medium” (Hamad, 1991). While this challenge is 
yet to be realized, the new age of scholarly communications immediately promises wide access at low cost. 
Furthermore, if scholars’ intention is to make their scholarship available to a wide audience, what better 
mode than the ejoumal? 

Greater Access to Scholarship 

The great pleasure of this new world of electronic publication comes at least once a day to those 
who edit and publish scholarly ejoumals. The same computers that "serve" out the publications also record 
the location (nation, for sure, sometimes the university or organization) of the recipient (the personal 
identity of the recipient is generally indeterminate, though not always). By scanning the daily access logs, 
one can see the locations of persons who visited the journal that day and downloaded articles. An 
established ejoumal. Education Policy Analysis Archives (EPAA) <http://epaaMsu.edu>, shows the breadth 
of access to scholarly writings that is far greater than that of competing journals. The access logs for EPAA 
for Febmary 14, 2000, revealed that people who visited the journal that day downloaded the following 
numbers of articles: 
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Location 


Number of articles 


US school districts 


159 


Canada 


154 


Spain 


28 


Sweden 


22 


New Zealand, (Aotearoa) 


20 


Argentina 


19 


Portugal 


19 


Brazil 


18 


Japan 


17 


United Kingdom 


15 


China 


13 


Singapore 


12 


Hong Kong 


12 


Netherlands 


12 


Taiwan 


11 


Mexico 


11 


Germany 


10 


Norway 


10 


Australia 


10 


France 


9 


Venezuela 


8 


Switzerland 


8 


Saudi Arabia 


8 


Romania 


7 


Turkey 


6 


South Africa 


5 


Colombia 


4 


Croatia (Hrvatska) 


4 


Egypt 


2 


Austria 


2 


Denmark 


2 


Peru 


2 


Philippines 


2 


Greece 


1 



Numbers like these are not simply gratifying, they are also very informative. The more than 642 
articles downloaded far surpass the logs of any library access to a journal in education policy in a month. It 
is difficult to imagine this number of such individuals making their way to the stacks of the nearest 
university library to find a research article on education policy. 

The connections from Argentina, Brazil, Croatia, Mexico, Peru, Philippines, Turkey, and 
Venezuela represent accesses to scholarly literature that has often been beyond the reach of these 
individuals, because commercial publishers charge libraries as much as $200 U.S. for Journals of fewer 
than 300 pages. The Journal, EPAA, competes for attention with three other Journals in its field whose 
combined subscriptions total about 5,000. One article in EPAA has been downloaded 25,000 times since it 
appeared in 1 995. A national survey of home schooling was published in the Journal on March 23, 1 999, 
and surpassed 9,500 downloads on August 1, 1999. On January 1, 2000, Linda Darling Hammond of 
Stanford University published a major report of research on teacher certification that was downloaded over 
5,000 times in the first five weeks. 

Commercial Interests 

Publishers of commercial for-profit print Journals charge different subscription rates to individuals 
and institutions. Due to the increase in cost for print materials, the subscription rates to these serials have 
risen dramatically. As a result, university libraries have had to re-examine the number of subscriptions their 
budgets could accommodate. Since 1986, the 121 members of the Association of Research Libraries (ARL) 
have spent 124% more to purchase 7% fewer serial titles (Walker, 1998). Each year libraries have been 
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forced to cancel some subscriptions in order to continue receiving other journals. This, in turn, causes a 
reduction in the number of subscriptions and publishers are likely to increase prices to stay in the business, 
which continues the cyclic crisis. 

Commercial publishers have long served as the principal providers of research scholarship. G. V 
Glass, (1999a), stated that scholarly publishers such as Elsevier, Springer, Kluwer, SAGE, and others, who 
publish literally thousands of books and Journals each year, by some estimates earn profits approaching 
40%. These publishers rely primarily on scholars to help with the review process, and these reviewers often 
present this service gratis. Most such publishers who offer electronic versions of their print journals also 
charge a subscription fee to access them. With the advent of web technologies, scholars and universities 
can take control of the publication process and disseminate research to all by providing free access. 

Gene V Glass, editor of Education Policy Analysis Archives (EPAA) <http://epaa.asu.edu/>, at 
Arizona State University stated: 

“I have published for seven years, completely by myself — no secretary, no graduate 

assistant, no budget--a peer refereed journal in education policy analysis that anyone can 

access for free on the intemef’ (1999a). 

Current Issues in Education (CIE) <http://cie.ed.asu.edu/> also published by the College of 
Education, at Arizona State University, since 1998, is supported with the salaries of two quarter-time 
graduate assistants and server space on one of the College’s servers. 

Limitations of Print Journals 

Print journals are captive to the temporal, geographic constraints of the predictable paper 
publication medium. Scholars submit their research findings to publishers as written reports, which are then 
sent out for peer-review. The peer-review process may take anywhere from three weeks to six months. The 
author makes revisions based on recommendations, and returns the manuscript to the editor(s). After the 
article is finally accepted, it again takes from six to twelve months or more before the published version 
appears. The entire process from the author’s initial submission to final acceptance and publication could 
take anywhere from a year to two years. Take for example, G. V Glass, 1998; the Educational Researcher 
received the manuscript on September 13, 1996; revision was received on April 7, 1998; accepted on May 
1, 1998; and finally published in November 1998 (p. 37). In this instance it took two years from manuscript 
to print and begin dissemination. This however, does not represent the end of the delay, for this represents 
the beginning of scholarly discussion. In order for other scholars to read, review, begin exploring the ideas 
presented, and perhaps offer responses it could take another couple of years, by when the author could be 
doing something else. This brings to focus the simple question, why do scholars publish? It is posited that 
scholars publish to share their creative ideas with others in the field and generate exchanges with peers. The 
rate at which the print medium engenders scholarly communication is one important reason why ejoumals 
are more practical than their counterpart. Thus ejoumals are helping redefine how scholarship is 
disseminated. 

The opportunity to create free access to scholarship via the Internet is unparalleled. Previously, 
researchers’ access to scholarship was limited to those who could afford subscriptions to print publications. 
The relative affordability of publishing an e-joumal opens access to all and allows for wider dissemination 
of knowledge. This represents the fundamental premise of ejoumals; democratization of access to 
scholarship and promotion of a global community of researchers (Leavy & Ganesh, 2000). 

Technological Affordances 

The power of web technologies lies in their ability to redefine research reporting and transform the 
nature of scholarly communication in ways not feasible in traditional print media. Increasingly, the Internet 
provides enhanced opportunities for visual, audio and video interactivity. The potential to incorporate 
features that advance or surpass those traditionally used in print journals is yet to be realized by authors and 
publishers of ejoumals. Web technology frees scholarly publications from size limitations imposed by the 
high cost of print. Ejoumals break the bonds of sequential physical publication. They can always be 
reformed to place related texts in close proximity. Additionally, the Internet delivers research data in a 
multiplicity of new formats. S. R. Glass, (1997), for example, in an analysis of autonomy in public and 
private schools, presents the entire data corpus comprising the full-text of thirty-seven interviews. 



133 



ERIC 



BESTCOPYAVAILABLE 




providing verifiability of assertions and confirmation of analytic integrity between researcher and reader. 
This mode of delivery, by making the record of data public for critical examination, fundamentally alters 
our mechanisms for establishing validity of reported research. The existence of the interview transcripts in 
their entirety allows the reader to function as a co-analyst with the researcher better and more critically than 
if he or she had to take the word of the author without ready access to the data. McLean (1997), utilized this 
public record to re-analyze S. R. Glass's data, using strict interpretations of rules of qualitative research. 

Similarly, limitations inherent in the use of summary statistics in research reports are an 
unavoidable reality due to space limitations imposed by the print medium. Russell and Haney (1997), 
include all of the raw data on which their analyses is based in two formats; text and spreadsheet. Dugan and 
Behrens’ (1998), use of embedded frames affords the reader interactivity and, as with McLean (1997), 
allows the reader to assess criteria for establishing validity of the analyses by avoiding data reduction, and 
offers extensive detail afforded through the use of hypertext and frames. The employment of hypertext in 
this exemplar facilitates instant and simultaneous access to multiple sources of information. The 
employment of frames allows access to the raw data, and the reader access to a variety of alternative 
models of data analysis without interruption to reading. 

In the field of education research, these inroads are rare. It is useful, therefore, to examine how 
other fields have taken advantage of the multiple modes of data representation and information sharing 
afforded by web-based technologies. The Journal of Seventeenth Century Music, for example, provides 
early, albeit promising, use of Internet technology to analyze audio. Silbiger (1996), compared the music 
genres Passacaglia and Ciaccona with text accompanying the written music and actual audio samples of the 
music (MIDI files). The audio feature is unique to electronic journals and is an indispensable element in 
certain fields of study such as music. It provides readers with insights to the author's arguments and allows 
one to make evaluations of the music. It is speculated that the comprehension of interviews and 
conversations in classroom research may be augmented by actual audio passages with accompanying 
commentary and analysis. Moreover, if transcripts are also included, the reader can assess the quality of the 
transcription and analytic method (Middleton, 2000). 

The challenge of using multimedia appropriately, powerfully, and not for its own sake, is 
demonstrated by Leshowitz (1999), who provided video clips to furnish readers a glimpse into the practices 
of a college classroom. The opportunity to witness the actual active-learning teaching methodology, student 
reactions, and changes in students' critical thinking skills that occurred during the course of the research 
study is afforded by providing a visual and auditory window into the reality of the classroom. While the 
choice of clips, and even the choice of what to video in the first place is arbitrary to some extent, judicious 
use of video has proven to be meaningful in this reporting context. Yet, these technologies imply that 
ejoumals have to face the dilemma of making innovative use while also ensuring equity and access to the 
visual and hearing impaired. Publishers of ejoumals themselves must be capable of applying modem 
technology and must facilitate these opportunities. 

Writing for ejoumals is challenging and scholars are yet to take full advantage of the capabilities 
that the medium offers. Features unique to the ejoumal should not distract from the flow of the information, 
yet these features should be used judiciously to enhance the important issues related to the topic of the 
write-up (Cesarone, 1 999). 

Furthermore, web technologies have advanced to provide full-text search capabilities. Print 
journals also provide search aids, such as indexing; none, however, match the capabilities of ejoumal 
search engines that facilitate searching with Boolean logic. Full-text searching of ejoumals as they continue 
their growth will soon be a universal feature. 

While most ejoumals in education differ little from print publication, an exploration of why this 
new technology has yet to change the fundamental nature of scholarly reporting is necessary. Scholars may 
not have access to the power the medium offers and expertise in use of cutting edge technology may rest 
with a few. 

Publishers of ejoumals, while using the unique features of the medium, have to take special care to 
ensure that the material is easily downloadable for all readers, including those who are on a modem 
connection. Web usability experts have shown that users are reluctant to wait for little longer than a few 
seconds to download information. 

Concerns that ejoumals are ephemeral can be alleviated by advances in network technology that 
offer mirror sites, actually enabling a geographically distributed digital archive (Ganesh & Jennings, 1 999). 
The Educational Resources Information Center (ERIC) Clearinghouse on Assessment and Evaluation 
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abstracts articles appearing in the Education Policy Analysis Archives in the Current Index to Journals in 
Education and permanently archives the entire contents of the journal in Resources in Education. 

Barriers to the Growth of Ejournals 

Concerns about the quality of ejournals and their scholarship are largely misplaced. Perceptions 
that ejournals are less rigorous are far from real. Brand, (1999), who interviewed several scholars from 
various disciplines, reported that educationists associated rejection rates and age of the journal as indicators 
of quality. Whereas scientists favored peer review over nothing at all; if they had to choose between timely 
access and peer review, many would choose timely access. The factor of speed with which ejournals could 
obtain peer reviews was actually an issue of concern to educationists, while scientists believed that peer 
reviews could work with both print and ejournals. Educationists felt that if ejournals were published 
quickly, they would lack rigorous peer-review and publishing standards. Researchers in education thus 
valued the referred review-revision negotiation process. 

Conclusion 

A redefinition of scholarly communication will significantly impact publishers, scholars, and 
libraries. Our concept of “a scholarly joumal” as established in the traditional world of print journals is 
under serious challenge due to recent advances in communication technology. Contemplation and 
clarification of what we value in scholarly publication, serious examination of the various aspects of 
electronic publishing and an understanding of their implications are necessary. The advantages and 
limitations of electronic journals for dissemination of scholarship should be clearly understood in order to 
apply available technologies to scholarly publication. Scholars can, and should, reclaim the publication of 
their works from commercial interests and make scholarly information freely accessible to the world. 
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Abstract 

Over the past decade, user interfaces for computer programs have become much more polished. 
Interfaces for computer-based instruction have similarly evolved. Today, the graphical and 
computational power of desktop computers makes intricate interfaces possible. Interfaces outside 
the area of instruction today are often metaphorical — expressing functions in terms of other 
things, anthropomorphic — putting a human face on the computer, or virtual— framing the user's 
interaction in terms of moving through a space. Applying these interfaces appropriately to 
learning tasks, the designer of computer-based instruction wields a powerful tool. This paper 
explores these three types of interfaces, including benefits they convey to learners, when their use 
is appropriate, principles for developing them best, and criticisms of their use. 



Introduction 

During the past decade and a half, computer interfaces have matured from command-line 
interaction to the graphical user interfaces of the Macintosh and Windows operating systems. Computer- 
based instruction matured and proliferated at the same time and grew quickly into the new operating 
systems, as demonstrated by the popularity of GUI authoring tools from the early Apple HyperCard to 
Macromedia Authorware and Asymetrix Toolbook. 

Inside and outside the instructional technology literature today, there is considerable discussion of 
user interface — how it should be designed, what forms it should take, and how it can improve learning 
(Bishop and Cates, 1996; Cates, 1996; Regian, Shebilske, and Monk, 1992; Thurman and Mattoon, 1994). 
Today, many instructional programs rely on metaphors to integrate their navigation, while new research is 
focused on virtual reality as a tool for simulation and human-like interfaces as enabling more natural 
interaction. 

The choice of interface is a powerful tool for the instructional designer; each type of interface has 
features that seemingly make it “better” suited for certain interactions, subject matter, and learners. 
Currently, however, there is little consideration of these possibilities in the design of computer-based 
instruction. 

This paper examines these three classes of interface and their potential benefits for learners. Each 
section will discuss the situations where their use is appropriate, principles for using them effectively, and 
criticisms leveled against them. 

Human-Computer Interface 

There are many options for the human-computer interface that can range from a simple and not too 
user-friendly design to those that attempt to use intelligence to anticipate the user’s need and appear human. 
Our interest in this paper is the metaphorical interface. A metaphorical interface is one that uses events, 
objects, and institutions from the real world (Erickson, 1991) as a framework for symbolizing, presenting, 
and organizing the functions of a computer program (Bishop & Cates 1994; Cates, 1994; Erickson, 1991). 
This type of interface presents user options in concrete terms to make the interaction easier and more 
efficient to the user. 
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Fisure 1. Advanced Dungeons and Dragons Core Rules CD-ROM 

The anthropomorphic and virtual interfaces are not models competing with the metaphorical 
interface but subsets of it. An anthropomorphic interface is one that uses the human being as its metaphor 
(Laurel, 1991). The computer’s feedback to the user, and to a greater or lesser extent the input from the 
user, is framed in terms of an interaction between two people. The most common example of this today is 
the Microsoft Office Assistant, which is given a semi-human appearance (depending on which assistant the 
user chooses, from Albert Einstein to a paper clip) and attempts to answer user questions typed in natural 
language. 

what would you like to do? 




Fi^re 2, Microsoft Office Assistant 

A virtual interface uses a spatial metaphor and allows three-dimensional movement or exploration 
(Regian & Shebilsky, 1 992; Thurman & Mattoon, 1994). For the purposes of this paper, an interface that 
incorporates a three-dimensional spatial model is considered virtual reality. An example is Microsoft’s 
Ultimate Frank Lloyd Wright, shown in Figure 3. Our discussion will be based on the underlying model of 
interaction and not limited to applications which require special input or output devices. An example of a 
virtual interface, then, could be as complex as NASA’s training and telepresence equipment (Fisher, 1991) 
or as relatively simple as a Web-based Virtual Reality Markup Language (VRML) site which requires no 
special hardware. 
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Fieure 3. The Ultimate Frank Lloyd Wright 



Metaphorical Interfaces 

The metaphorical interface is the superset of all the interfaces discussed in this paper. 
Anthropomorphic and virtual interfaces are specific types of metaphorical interfaces popular in the 
literature. In this section, the metaphorical interfaces discussed are specifically not anthropomorphic or 
virtual. 

Figure 4 illustrates the interface metaphor familiar to most computer users, the file folder 
metaphor of Microsoft Windows (as well as the Apple Macintosh). Files are organized for the user in 
metaphorical folders (the actual files are not organized in the same way). Visual icons, metaphors in 
themselves, represent individual files. 
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Fi 2 ure 4. Microsoft Windows interface 

In computer-based instruction, Cates (1996) identifies two types of metaphor: the primary 
metaphor is the overall metaphor, providing structure to the instruction, while auxiliary metaphors are 
additional metaphors to support and enhance the primary one. For example, in the Dungeons and Dragons 
Core Rules screen shown earlier in Figure 1, the primary metaphor is that of a game player’s desktop. 
Objects on the desktop are secondary metaphors; the miniature knight can be clicked on for character 
generation, the globe for a mapping program. 
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Benefits to the Learner 

According to Bishop and Cates (1996), the interface metaphor offers two essential benefits to the 
user: it establishes expectations for how things work and it provides a link to previous knowledge. 
Metaphors establish expectations by phrasing an idea in terms of another, more familiar idea; they link 
things based on functional characteristics and thus give users an idea of what a particular button or 
command will do. As such, they make learning to operate the system easier; in instructional software, in 
particular they speed the process of navigation. Second, metaphors link to prior knowledge. They can 
explain a new idea in terms of an already known one, and so integrate into learners’ existing schema easily. 
Metaphors help students learn by pointing out similarities (Bishop and Cates 1996). For example, the folder 
icon in the operating system represents a place to store documents much like a real folder in a file cabinet. 
This metaphor provides a concrete, realworld correlate for the more abstract term of directory. 

When to Use a Metaphorical Interface 

Metaphor is a basic function of human thought and communication (Bishop and Cates 1996; Cates 
1996; Erickson 1990;), which suggests that its use will be beneficial to the user. The use of a metaphor, 
then, is generally based on the designer’s ability to develop an interface scheme using a helpful metaphor. 
The usability criteria discussed in the next section provides a basis for deciding whether or not to use a 
particular metaphor. 

Design Principles for Metaphorical Interfaces 

Erickson (1991) suggests a three-step process for designers to design an interface metaphor. First, 
identify the software function to be signified, and potential user problems with understanding it. Second, 
examine the metaphors already implicit in the function; an interface metaphor must amplify these, not 
contradict them. Third, with this process in mind, the designer must look for real-world institutions, 
objects, or events that possess similar characteristics. As an example, we might hypothesize how the 
designers of the Eudora email program selected the trash can as icon for the delete button. First, the 
software function was one of deleting, erasing, or throwing away an email message. Second, the designers 
had to consider metaphors that conveyed the process of deleting, erasing, or throwing away. One obvious 
possibility was that of an eraser or pencil with an eraser. However, that icon is often used in graphics 
programs to alter part of an image. Third, the designers might arrive at the idea of a trash can as it 
symbolizes throwing something away and removing it permanently. 

Erickson (1990) also proposes evaluational criteria for metaphors. The designer should measure 
the amount of structure a metaphor provides, the applicability of the structure to the function at hand, the 
imagery-producing capabilities of the metaphor, the suitability of the metaphor for the target audience, and 
use of the metaphor elsewhere in the program and in the future. Not only are these criteria for an 
appropriate metaphor at the time of selection, they are criteria for testing and final implementation of the 
metaphor. A good design takes the metaphor to its fullest extent along these lines. 

Criticisms of Metaphorical Interfaces 

The most common criticism of metaphors in interface is that poor metaphors reduce the usability 
of a system. In learning applications, ineffective metaphors can cause cognitive dissonance and a reduction 
in learning. Erickson (1990) provides an example of the answering machine as an inappropriate metaphor 
for a voice-mail system; while the two share an important feature, message-taking, there is an essential 
difference, in that voice-mail is not typically located near the user, while an answering machine is 
physically present. This separation makes some aspects of the system extremely difficult to understand — 
such as when a message takes a half-hour to reach the user. It is important to recognize, however, that this 
is not a criticism of the use of metaphor, but of specific implementations. 

Nelson (1991) offers two additional criticisms. First, a metaphor inhibits the production of new 
ideas, instead focusing the designer on extending known functions. That is, an existing metaphor imposes a 
structure on the designer. Second, metaphors become dead weight, forcing everything in the program to fit 
even when it does not by nature. The result is metaphors are often forced far beyond their capacity. This 
problem becomes apparent in the desktop metaphor, from the early Xerox PARC and Apple Macintosh 
implementations through essentially five generations of Microsoft Windows. New functions were forced to 
fit into the existing metaphor, making them difficult to design and to comprehend. 
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Anthropomorphic Interfaces 



An anthropomorphic interface, as defined in the introduction, is one which uses the human being 
as its defining metaphor. Figure 2 already illustrated Microsoft’s Office Assistant, a feature of Microsoft 
Office 97 (with close parallels in Office 98 for the Macintosh and Office 2000). The Assistant frames the 
interaction between the user and the computer as a conversation, with the user asking questions and the 
assistant suggesting tips. Figure 5 illustrates the High Council in Microprose’s game Civilization II. The 
Council has five members, each of whom comments on the user’s performance in different areas. Figure 6 
demonstrates the Web site Ask Jeeves ( http://www.askieeves.com) . which also frames the interaction as a 
conversation. The user poses a natural English question to the agent, a meta-search engine, which responds 
in the first person, “These are questions I know the answer to,” suggesting an approximation of the user’s 
search. The interface is given a static human appearance as well, through an illustration of a butler. 





:^^u ask^|;;?|compL|ter Interface design . 
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Fisure 6. Ask Jeeves 

Benefits of Anthropomorphic Interface to Learners 

Many of the benefits of anthropomorphic interfaces are assumed; the literature presents few 
examples of a proven statistical benefit in learning from an anthropomorphic interface. However, many 
benefits have been deduced from anthropology and psychology. 

One benefit commonly cited is that, as human beings, we are good at communicating with other 
people; we do it naturally from an early age, if not instinctively. (Brockmann, 1997; Laurel 1991). A well- 
designed anthropomorphic interface can take advantage of this communication skill and lower cognitive 
load allowing users to communicate in a kind of “cognitive shorthand” that requires less time to learn a 
new system (Laurel 1991, Oren, Salomon, Kreitman, & Don, et al, 1991). 

Second, there is a motivational element to anthropomorphic interfaces. They can implement a 
storylike structure to an educational program, one that leads the user; their richness can be almost 
“seductive”. They can create a suspension of disbelief and an overall atmosphere of engagement for the 
user (Oren et al., 1991). 
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Some authors argue that a humanlike interface is simply a fitting metaphor for a computer 
program: like humans, computer programs are responsive to communication and capable of carrying out 
tasks. Moreover, the appearance of a human or similar character can draw the user’s attention to features of 
the software. For instance, Apple Computer gave their information agent Phil a bow tie to accent his role as 
the user’s personal assistant or butler (Laurel, 1991). 

Finally, anthropomorphic interfaces can provide the capability for the user to personalize the 
interface in terms of appearance, characteristics, or content presented (Brockmann, 1997; Griffin, 1996; 
Oren et al., 1991). 

When to Use an Anthropomorphic Interface 

Few authors make suggestions on when it is appropriate to use anthropomorphism in interface 
design. Laurel (1991) is an exception, offering a suggestion of tasks an anthropomorphic software agent 
might perform, from information retrieval to tutoring to playing against the user in a game. Laurel also 
offers a general heuristic for applying anthropomorphic interfaces: they should be employed when the task 
is too complex for simple algorithmic solution or complete parameter specification by the user. This applies 
not only to the interface, however, but the expert system or other type of programming that would support 
solving the task. 

Design Principles for Anthropomorphic Interfaces 

The literature is not nearly as silent on how to apply anthropomorphic design. A number of 
principles are offered. First, in answer to some of the criticisms examined below. Laurel (1991) suggests 
that an anthropomorphic agent must pass an ‘‘Anti-Turing” test. A Turing test is a measure for artificial 
intelligence: a computer passes the Turing test when the user thinks he has interacted with another human 
being, not a computer. Thus, Laurel stipulates that it should be clear that the anthropomorphism is indeed 
part of a computer program, to cut down on user confusion. Using familiar or dramatic actors that are 
clearly recognized by learners as part of an external reality, for instance, helps reduce anthropomorphism- 
based confusion. Using cartoonlike characters would fulfill a similar function. 

Similarly, Oren et al. (1 991) suggest that the instructional designer provide a framework for the 
learner that makes it clear that the user is dealing with a computer program. In their research, they had the 
actress who provided the interface guide provide an out-of-character introduction to the user. This addition 
made the interaction clear as a sort of “make-believe” session. 

Brockmann (1997) presents five characteristics, all part of the interaction as a form of cooperation, 
for good anthropomorphic interface: 

• Forgiveness of user mistakes 

• Nonauthoritarian presentation, using suggestions instead of mandates 

• Prediction of user behavior and needs 

• Basing actions on the context of the current user and recent sessions 

• Adaptation to different users 

These principals are an abstraction of human conversation to possible human-computer 
conversation. 

Criticisms of Anthropomorphic Interfaces 

The anthropomorphic interface has invited attack from several quarters. These criticisms focus 
primarily on anthropomorphic features of the design. 

The first argument is that a human appearance for the computer is inappropriate to human- 
computer interaction. For example, Brockmann (1997) cites a study in which students preferred less 
humanlike feedback in computer-based learning instead of a more teacher-like presentation by the 
computer. Another criticism is that it is difficult to make a personality or even a gender fit a computer 
program, which could lead to user confusion, dissatisfaction, or frustration (Brockmann, 1997). 

Second, an anthropomorphic interface can be misleading. It can make the computer seem too 
human and risks user backlash in response to this dishonesty (Brockmann, 1997). 
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Third, some critics fear that transferring human characteristics to a computer will result in users 
transferring computer characteristics or aspects of their interaction to other humans. The result could be an 
undervaluing of humans or abuse of people in a serving capacity (Brockmann, 1997; Laurel, 1991). 

Implementing an anthropomorphic interface also introduces an indirectness that raises concerns 
(Laurel, 1991). Instead of a user interacting directly with parameters or functions, the user has to frame 
activity in a conversational context. This added cognitive activity can lead to less productivity and user 
frustration. Many users have experienced this with the Microsoft Office Assistant; particularly for power 
users, it is a much slower means for obtaining help than the traditional Windows help interface. Finally, 
anthropomorphic interfaces can introduce an irritation factor based on their appearance or functionality. 
Again, experienced users may find the constant interruption of the Assistant in Microsoft Office an 
irritation. 

None of these criticisms provide an argument suggesting one should not use an anthropomorphic 
interface. To the “inappropriate context” argument, one could counter that that*s a design decision based on 
learner and environment analysis. Against the “misleading interface” and “devalued human” interfaces 
comes the design principle that the computerized nature of the interface should be made completely clear, 
reducing or eliminating spillover to non-computer interactions. 

The indirectness argument is more problematic; however, in many cases this problem is another 
analysis-and-design issue. Anthropomorphic interfaces could enhance the instructional environment for 
some learners while detracting from the instruction for others. More research is needed to determine what 
type of learner, if any, benefits from the interface. An alternative is to provide an option for the learner to 
use a more standard interface or to opt for a metaphorical interface. Such an approach would also address 
the problems that result due to the “irritation” argument. 

Virtual Interfaces 

The fundamental characteristic of a virtual interface is its three-dimensional, spatial nature. Virtual 
interfaces could be exact representations of physical phenomenon in existence or possible, or they could be 
spatial representations of essentially abstract data (Fisher 1991). 

Figure 7 illustrates Piranha Interactive’s Origins, which offers several “virtual reality 

reconstructions” of ancient buildings. The learner can navigate through the buildings with the mouse cursor 
and get more information in particular areas. Figure 8 shows a more abstract virtual space, Financial 
Viewpoints. The program uses a virtual space to organize information for the learner. 
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Benefits to Learners 

The benefits of virtual reality or other virtual interfaces have been widely discussed in the 
literature. Some benefits are assumed and have little empirical support; however, there is a growing body of 
research demonstrating the effectiveness of virtual interfaces to create spatial learning (e.g., Regian and 
Shebilske 1992). 

The broadest benefit suggested for the virtual interface is that it allows for multi-channel input 
from and output to the learner. Depending on the hardware and software used, the technology could take 
input in the form of voice commands, eye movements, hand gestures, or other natural human 
communications. In return, it can enhance its output with stereo sound giving spatial cues to the human 
auditory system, three-dimensional illustrations, and tactile response (Fisher 1991). This enhancement can 
create an immersive environment which is motivating, suspends disbelief, and takes advantage of human 
perception far better than non-virtual interfaces (Thurman and Mattoon 1 994). 

As an intrinsically spatial medium, virtual presentation preserves the visual and spatial 
characteristics of what it presents. This fidelity allows the interface to present more information and 
reduces students' cognitive load as it removes the need for them to construct their own representations from 
two-dimensional or even textual data. The representation presented by the interface can supplant the 
mediation the learner requires to comprehend the information. 

Finally, a rich virtual environment allows for interaction between the learner and everything 
simulated and can create emergent results based on this interaction. This interaction creates a more lifelike 
experience for the learner (Thurman and Mattoon 1994). 

When to Use Virtual Interfaces 

There is limited information on the appropriateness of the virtual interface in the literature. Regian 
and Shebilske (1992) offer some suggestions: A virtual presentation is appropriate for spatial data, the 
practice of procedural tasks, reinforcement of verbal skills through imagery, and navigational tasks. 

Thurman and Mattoon's (1994) highlighting of a virtual interface's ability to aid visualization of 
invisible data such as wind and gravity suggests another avenue for successful implementation of a virtual 
interface. Virtual interface may aid the learning of invisible phenomena that can be represented graphically 
and spatially. This suggestion is supported by Dennen and Branch’s rule that virtual reality should be used 
for presenting spatial or abstract data (1995). 

Design Principles for Virtual Interfaces 

Few heuristics for the design of virtual-interface learning or training have been published. Dennen 
and Branch (1995) suggest several, however, from their survey of virtual reality designers and instructional 
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technologists. It is important to note, however, that these suggestions are rules-of-thumb and are not 
presented with any empirical support 

David Lebow presents five constructivist values for instructional systems design (Lebow, 1 993), 
By the virtual interface’s nature as an open-ended environment allowing and encouraging exploration, it 
supports three of these. The first value is protecting the learner from harmful effects of instructional 
practices; one of the directions Lebow identifies for instructional designers is to balance controlling the 
learner with the promotion of personal autonomy. The second value supported by virtual interfaces is 
embedding the reasons for learning into the activity. Virtual interfaces present an opportunity for the 
instructional designer to provide a rich context for learning and practice, answering Lebow’s call to let 
learners practice in environments close to the application environment. Finally, Lebow also suggests 
letting learners explore their errors. A virtual application can provide this kind of practice for learners 
without those errors having a lasting effect in the real world. Further, while Lebow points out that mistakes 
have strong negative connotations, by removing the learner a step from reality, virtual environments may 
remove some of this pressure on the learner. 

Dennen and Branch suggest five other characteristics of a good virtual interface: 

• It should be user-friendly 

• It should allow and even require a high degree of user action 

• It should offer built-in tools for interaction and exploration 

• It should allow teachers to interact in the environment 

• It should allow user experimentation and expression ( 1 995) 



Criticisms of Virtual Interfaces 

The relative newness of the virtual interface in the instructional technology literature has resulted 
in few criticisms. The lack of significant research on virtual interfaces presents the designer with a 
dilemma of both how and when to use this type of interface. A second concern is the transferability of skills 
from the virtual environment to the real world. Third is a concern of the degree of fidelity that is needed to 
create an appropriate and effective interface. 

Donald Norman offers a major criticism of virtual reality as a solution to interface problems. 

There is a difference between spatial ability and visual ability, and virtual reality, he asserts, as presenting 
pictures of spaces relies on the latter, 

[WJhat most technologists seem to want to do is present a picture of a three-dimensional world on 
the screen. . . But there is no movement; the visual world moves, but we ourselves stand still. That is not at 
all the same as the real situation in which the world stays still and it is we who move, . , The experience is 
not that of movement (Norman, 1998: 101). 

Norman recognizes the value of virtual reality as a training tool but warns against incautious 
acceptance of it as a solution. The criticism he offers merits further study: How close is the experience of a 
virtual interface to the experience of a true space? 

Conclusion 

The following is a set of heuristics derived from this research; the most reliable are those which 
simply suggest when each type of interface is most appropriate: 

• A metaphorical interface in general is useful for organizing a user’s navigation through 
the learning application. 

0 An anthropomorphic interface is useful when the computer is performing humanlike 
functions: coaching, guiding, analysis, opposing the learner in a game, and so on, 

• A virtual interface is useful when teaching spatial or visual information, or abstract 
information that lends itself to a spatial or visual representation. 

A review of this literature on this topic raises two concerns. First, the literature is often not 
empirically based and deals more with possibilities for different interfaces than what is necessary for 
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instruction with them. Several articles do present studies with quantitative data supporting some of their 
statements; however, many of the articles are not oriented towards verification using either a quantitative or 
qualitative approach. Much of it is instead anecdotal: “We did this and it worked/’ This type of article 
might be classified as preresearch (Morrison & Adcock, in press). 

Finally, much of the discussion is focused to what the interfaces can be used for, and not what 
aspects are necessary for learning. A study of the aspects at play in virtual interface learning, or learning 
with any of the other interfaces, could lead to design principles for instruction in different media. The 
research, however, has not adequately addressed this issue. 

The articles discussed here raise hope, however. They provide an overview of thinking and 
application in the field in the past decade and so suggest avenues for more experimental research. In their 
suggested benefits and design principles they provide hypothesis to be verified through careful testing. As 
the power of computers increases, making some of the more complex aspects of the interfaces discussed 
here possible, and as computer-based learning continues to mature, this kind of research will become 
increasingly valuable. 
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Overview 

This paper reports on a needs assessment conducted to answer the following questions: (I) What 
is the optimal instructional content for a foundations course in Instructional Technology? (2) 
What is the optimal delivery method for a foundations course in Instructional Technology? (3) 
What feelings do respondents have about the use of the Internet for a foundations course in 
Instructional Technology? Methods and techniques for the needs assessment followed suggestions 
provided by Rossett (1987) in her book, Training Needs Assessment. 



Data Sources 

Extant Data. 

We collected extant data to identify the content and topics most frequently covered in foundations 
or introductory survey courses in Instructional Technology. These data were obtained by examining syllabi 
for graduate-level courses taught at the following institutions - Arizona State University, Florida State 
University, University of Georgia, Indiana University, San Diego State University, and Syracuse 
University. 

Current Students. 

We contacted 35 students enrolled in our Instructional Technology program about participating in 
the needs assessment. Twenty-three students responded to the request indicating a 66% response rate. The 
majority of these participants were female (74%), between 23 and 30 years old (52%), and were seeking a 
master’s degree in Instructional Technology (87%). Most rated their level of computer skill as either 
intermediate (48%) or advanced (39%). Nine current students (39%) indicated previous experience with 
courses delivered via the Internet. 

Graduates. 

We also contacted ten graduates of our program about participating in the needs assessment. Eight 
of the ten graduates responded to the request indicating an 80% response rate. The majority were female 
(75%), over 31 years old (88%), and held a master’s degree in Instructional Technology (88%). Most rated 
their level of computer skill as intermediate (50%) or advanced (50%). Three graduates (38%) indicated 
previous experience with courses delivered via the Internet 

Faculty. 

We contacted ten individuals with faculty positions at Instructional Technology programs 
throughout the United States to request their participation in the needs assessment. Nine out of ten faculty 
responded to the request, indicating a 90% response rate. The majority of these faculty were male (78%) 






and over 41 years old (88%). Three faculty (33%) indicated previous experience teaching courses delivered 
via the Internet. 

Instruments. 

Each respondent group (students, graduates, faculty) completed a survey developed to address 
issues related to content, delivery method, and use of the internet for a foundations course in Instructional 
Technology. These surveys were designed to collect specific information from each respondent group. 
Items related to content were written based on a document analysis of the book Instructional Technology: 
Past, Present and Future by Anglin (1995). We developed an initial content topic list using the section 
headings from this book. Next, the list was expanded to include other topics that appeared in at least two of 
the course syllabi that we had collected. Respondents were asked to identify the topics that should be 
included in a foundations course from a list of 1 8 possible topics. Items related to delivery provided a list 
of five possible methods and asked respondents to indicate the optimal delivery method for a foundations 
course. Items related to feelings about use of the Internet asked respondents to indicate whether required 
use of this medium would encourage or discourage faculty/student interaction and help or hinder learning. 
We also collected data on participant demographics and experience with Internet and related technologies. 

Results 

Optimal Content. 

A major purpose of our needs assessment was to identify the optimal instructional content for a 
foundations course in Instructional Technology (IT). We examined the syllabi obtained from the six 
graduate programs listed above to identify the content topics most frequently covered in these courses (see 
Table 1). We found that (a) topics related to the definitions of IT and instructional design models were 
included in all six of the courses; (b) topics related to the history of IT and professional 
competencies/issues were included in five of the courses; (c) topics on evaluation, instructional 
theory/planning, learning theory, and trends in IT were covered in four of the courses; (d) topics related to 
innovation/change, needs assessment, and performance technology were included in three courses; (e) 
topics about adoption/diffusion of IT, distance education, Gagne's events of instruction, instructional 
message design, and media research were covered in two of the courses. 
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Table I: Topics covered by two or more Instructional Technolozv (IT) courses 



Topic 


Frequency 


Definitions of IT 


6 


Instructional Design Models 


6 


History of IT 


5 


Professional Competencies & Issues 


5 


Evaluation 


4 


Instructional Theory & Planning 


4 


Learning Theory 


4 


Trends in IT 


4 


Innovation & Change 


3 


Needs Assessment 


3 


Performance Technology 


3 


Adoption & Diffusion of IT 


2 


Distance Education 


2 


Gagne's Events of Instruction 


2 


Instructional Message Design 


2 


Media Research 


2 



We also asked graduates and faculty to identify the topics that should be included in a foundations 
course from a list of 1 8 possible topics. Table 2 provides a list of optimal course topics selected by 
approximately half or more of the total respondent group. This list includes - definitions of IT (100%), 
instructional design models (93%), trends in IT (93%), history of IT (80%), needs assessment (73%), 
instructional theory/planning (53%), professional competencies/issues (53%), adoption/diffusion of IT 
(47%), and evaluation (47%). 
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Table 2: Optimal course topics identified by graduates and faculty 



Course Topic 


Graduates 


Faculty 


Total 


Definitions of IT 


8 


7 


15 




(100%) 


(100%) 


(100%) 


Instructional Design Models 


8 

(100%) 


6 

(86%) 


14 

(93%) 


Trends in IT 


7 

(88%) 


7 

(100%) 


14 

(93%) 


History of IT 


6 

(75%) 


6 

(86%) 


12 

(80%) 


Needs Assessment 


6 

(75%) 


5 

(71%) 


11 

(73%) 


Instructional Theory & Planning 


5 

(63%) 


3 

(43%) 


8 

(53%) 


Professional Competencies & Issues 


5 

(63%) 


3 

(43%) 


8 

(53%) 


Adoption & Diffusion of IT 


3 

(38%) 


4 

(57%) 


7 

(47%) 


Evaluation 


3 

(38%) 


4 

(57%) 


7 

(47%) 



We examined the data on optimal course topics separately for graduates and faculty. Results 
indicated that the following topics were identified by half or more graduates - definitions of IT (100%), 
instructional design models (100%), trends in IT (88%), history of IT (75%), needs assessment (75%), 
instructional theory/planning (63%), and professional competencies/issues (63%). Topics identified by half 
or more graduates but not by half or more faculty were distance education (50%), Gagne’s events of 
instruction (50%), and media selection (50%). 

Turning to results for faculty, seven out of nine faculty responded to items regarding optimal 
course content. Results revealed that the following topics were identified by half or more of faculty who 
responded: definitions of IT (100%), trends in IT (100%), instructional design models (86%), history of IT 
(86%) needs assessment (71%), adoption/diffusion of IT (57%), and evaluation (57%). Topics identified 
by half or more faculty but not by half or more graduates were learning theory (57%) and media research 
(57%). It is interesting to note that of the two faculty who declined to respond to the items regarding 
optimal course content, one questioned whether the foundations of IT should be taught as a distinct course. 
The other indicated that topics could not be identified without knowledge of the objectives of the course. 

Optimal Delivery Methods. 

Another purpose of our needs assessment was to identify the optimal delivery 
method for an IT foundations course. Respondents were asked to indicate the optimal 
delivery method from a list of five possible methods (see Table 3). 
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Table 3: Optimal delivery methods identified by students, graduates, and faculty 



Course Delivery Method 


Students 


Graduates 


Faculty 


Total 


Entirely face-to-face 


7 


0 


2 


9 


classroom activities 


(33%) 


(0%) 


(28%) 


(25%) 


Emphasize face-to face classroom 


6 


5 


3 


14 


activities with online assignments 


(28%) 


(63%) 


(43%) 


(38%) 


Half online activities and half 


4 


1 


1 


6 


classroom activities 


(20%) 


(12%) 


(14%) 


(16%) 


Emphasize online activities with 


4 


1 


0 


5 


classroom meetings 


(20%) 


(12%) 


(0%) 


(14%) 


Entirely online 


0 


0 


0 


0 


(class never meets face-to-face) 


(0%) 


(0%) 


(0%) 


(0%) 


Other 


0 

(0%) 


1 

(12%) 


1 

(14%) 


2 

(5%) 



Twenty-one students responded to the items on optimal delivery method. Seven students (33%) 
indicated that the course should be delivered using entirely face-to-face classroom activities. Six students 
(28%) responded that face-to-face classroom activities with online readings and assignments would be the 
optimal delivery method. Four students (20%) indicated that the course should be delivered using half 
online activities and half face-to-face classroom activities. Four others (20%) responded that a course 
emphasizing online instruction with classroom meetings would be optimal. None of the students indicated 
that the course should be delivered entirely online. 

Eight graduates answered the items concerning optimal course delivery method. Five graduates 
(63%) indicated that face-to-face classroom activities with online readings and assignments would be an 
optimal delivery method. One graduate indicated that the course should be taught using half online 
activities and half face-to-face classroom activities, while one another stated a preference for an emphasis 
on online instruction with classroom meetings. In addition, one other graduate responded that one-third of 
the course should be online. None of the graduates indicated that the course should be delivered entirely 
online. 

Seven faculty answered the items concerning optimal course delivery method. Three faculty 
(43%) responded that face-to-face classroom activities with online readings and assignments would be the 
optimal delivery method. Two others (28%) indicated that the course should be delivered using entirely 
face-to-face classroom activities. Another faculty member indicated that the course should be taught using 
half online activities and half face-to-face classroom activities. The other faculty respondent included a 
statement that the class should be web supported, but the level to which the class should be online 
depended on the goals of the course. None of the faculty indicated that the course should be delivered 
entirely online. 

Feelings about use of the Internet. 

The final purpose of our needs assessment was to determine how students, graduates, and faculty 
felt about using the Internet for a foundations course in IT. Items related to feelings about use of the 
Internet asked respondents to indicate whether required use of this medium would encourage or discourage 
faculty/student interaction and help or hinder learning (see Table 4). 
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Table 4: Participant feelings toward use of the Internet 





Students 


Graduates 


Faculty 


Total 


In your opinion, required use of the 
Internet for the course would: 

Help students stay current with 


11 


7 


5 


23 


technology 


(48%) 


(88%) 


(71%) 


(61%) 


Prepare students for the job market 


6 


5 


5 


14 


(26%) 


(63%) 


(71%) 


(37%) 


Encourage faculty-student contact 


4 


2 


2 


8 


(17%) 


(25%) 


(28%) 


(21%) 


Encourage student-student contact 


4 


2 


1 


7 


(17%) 


(25%) 


(14%) 


(18%) 


Discourage faculty-student contact 


14 


3 


4 


21 


(61%) 


(38%) 


(57%) 


(55%) 


Discourage student-student contact 


13 


3 


4 


19 


(57%) 


(38%) 


(57%) 


(50%) 


Prevent some students from 


11 


4 


3 


18 


succeeding from this class 


(48%) 


(50%) 


(43%) 


(47%) 


Create a hardship for students 


6 


3 


3 


12 


(26%) 


(38%) 


(43%) 


(32%) 


Put an unfair emphasis on 


6 


2 


0 


8 


computer literacy 


(26%) 


(25%) 


(0%) 


(21%) 


Be inappropriate for this course 


0 


1 


2 


3 


content 


(0%) 


(13%) 


(28%) 


(8%) 



Twenty-three students completed the items concerning feelings about required use of the Internet 
for the course. Results revealed that (a) fourteen students (61%) indicated that the Internet would 
discourage faculty-student contact; (b) thirteen students (57%) responded that Internet use would 
discourage student- student contact; (c) eleven students (48%) felt that the Internet would prevent some 
students from succeeding in the course, and (d) six students (26%) felt that Internet use would create a 
hardship for some students and put an unfair emphasis on computer literacy instead of course content. 
However, eleven students (48%) thought that required use of the Internet for the course would help them 
stay current with technology and six students (26%) felt Internet use would prepare them for the job 
market. 

Eight graduates completed the items concerning feelings about required use of the Internet for the 
course. Seven graduates (88%) indicated that Internet use would help students stay current with technology 
and five graduates (63%) felt that such a requirement would prepare students for the job market. However, 
four graduates (50%) thought it would prevent some students from succeeding in the course and three 
graduates (38%) indicated that required internet use would discourage faculty-student and student-student 
contact. 

Seven faculty responded to the items concerning feelings about required Internet use. Five faculty 
(71%) indicated that it would help students stay current with technology and prepare them for the job 
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market. However, four faculty (57%) indicated that required Internet use would discourage faculty-student 
and student-student contact. 

Implications 

The purpose of this needs assessment was to identify the optimal content and delivery method for 
a foundations course in Instructional Technology and to determine the feelings of respondents toward the 
use of the Internet for this course. The results can be used to design or revise the introductory, survey 
course in Instructional Technology. 

At Arizona State University, we have begun to use the data from the needs assessment to identify 
the gaps between actual and optimal course content and delivery methods. For example, “Trends in 
Instructional Technology” was identified as an important topic by a large majority of respondents. 

However, our foundations course does not currently include this topic. We plan to include the topic the 
next time the course is offered. Furthermore, the current course does not include any form of on-line 
delivery methods. Results of our needs assessment indicated that sixty-eight percent of all respondents 
favored a delivery method that included some form of on-line delivery. Most of these respondents felt that 
the optimal delivery method should emphasize face-to-face classroom activities with readings and 
assignments on-line. 

While the needs assessment suggested that the course should include some form of on-line 
delivery, data also indicated that respondents had mixed feelings about the use of the Internet. Over half of 
all respondents felt that required use of the Internet for the course could discourage faculty-student and 
student-student contact and a third felt that it could create a hardship for some students. The high level of 
concern uncovered by this needs assessment suggests that implementation of on-line instruction should be 
accompanied by procedures designed to address these perceptions. 
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The Validation of a Measurement Instrument: 
Teachers’ Attitudes toward the use of Web Resources 
IN THE Classroom, across Two Cultures 



A positive attitude toward innovation is indicative of a mental state of readiness to sustain or 
adopt that innovation. This paper describes an instrument developed to measure teachers* 
attitudes toward the use of web resources in the classroom. A factorial analysis revealed an 
estimated alpha internal consistency score of .90 for a sample of United States teachers, .65 for a 
groups of teachers from Thailand, and an overall estimated alpha internal consistency score of 
.85. Collecting reliable data from teachers, with regard to their attitude toward the use of web 
resources in the classroom, can provide valuable insights into assessing readiness to integrate 
web resources into the classroom and interventions designed to help teachers enhance their 
teaching using web resources. This instrument was designed to collect such data. 

Introduction 

Powerful new technologies exist that can enhance instruction and learning, yet these new world 
wide web technologies are not being widely used by public school teachers (Wiburg, 1997). Teachers are 
not readily choosing to incorporate web resources into their teaching even though they have been 
empirically shown to facilitate interactivity (Hardin & Ziebarth, 1995; Owston, 1997), increase mental 
activity (Conlon, 1997), and promote social interaction (Kagan & Widaman, 1987; Sharan & Kussell, 

1984; Slavin, 1983) when used in the classroom. The plausibility of the argument that web resources are 
not used in classrooms because web technology is not available is quickly diminishing as web technology 
initiatives explode (Becker, 1998). Why then, are teachers not using these resources in the classroom? A 
possible explanation may be related to the teacher’s mental state of readiness to adopt this new innovation. 
This state of readiness is the teacher’s attitude toward the use of the web resources in the classroom. 

The Role of Attitude in Adopting Innovations 

Rogers (1995) suggested that the first step in adopting an innovation, such as web technology, was 
to be exposed to information and developing knowledge about the innovation. Teachers only need watch 
television commercials to see many examples of school children communicating with other students around 
the world or participating in conversations with astronauts through Internet communication networks. 
Billboards, advertisements, magazines and news articles are littered with references to web addresses 
demonstrating the vast amount of resources accessible through web technology. Teachers see technology 
invading their classrooms and attend in-service programs on use of the web resources for teaching. Thus, 
teachers are being exposed to many illustrations of the use of web resources for education. This exposure 
promotes the formation of various levels of knowledge about the value of web resources to teaching and 
learning. According to Rogers (1995) this knowledge promotes the formation of attitudes toward the use of 
innovations, in this case the use of web resources for classroom teaching and learning. 

There is much value in having a measure of teachers’ attitudes toward the use of web resources in 
the classroom. These data could be helpful in assessing teacher readiness to incorporate new resources into 
the classroom. They can also be used to inform the development of interventions designed to better inform 
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teachers about the flexibility and value of different types of web resources for the teaching and learning 
process. Yet, no instrument could be found that specifically measures teachers’ attitudes toward the use of 
web resources in the classroom. The purpose of this paper is to report on the development and factorial 
validity of an attitude instrument entitled Teachers’ Attitudes Toward the Use of Web Resources in the 
Classroom. 

Constructs of Attitude 

Thomas & Znaniecki (1918) defined attitude as “a mental and neural state of readiness, organized 
through experience, exerting a directive or dynamic influence upon the individual's response to all objects 
and situations with which it is related.” Zimbardo & Leippe (1991) more recently defined attitude as an 
evaluative disposition toward some object based upon cognitions, affective reactions, behavioral intentions, 
and past behaviors. Attitude is an informed predisposition to respond and is comprised of three constructs: 
(1) cognitive - belief, (2) affective - feeling, and (3) behavioral - a readiness or intent for action (Zimbardo 
& Leippe, 1991). 

Lawless & Smith (1997) defined the belief construct as a trust, confidence, or perception that 
something, such as web resources, would provide benefits. For example, “effectively incorporating web 
resources into lessons can enhance instruction” assesses the teacher’s level of agreement that web resources 
provide the benefit of enhanced instruction. 

Feelings are defined as an emotional, affective, or internal response toward or away from 
something that is generally predicated on participatory experience (Fishbein & Ajzen, 1975; Zimbardo & 
Leippe, 1991). Thus, being asked to indicate an opinion about web resources by identifying with opposite 
terms such as “bad and good” invoke internal feelings about using web resources, not tied to benefits but as 
an emotional response. 

The third construct of attitude is behavioral or intent toward behavior. This construct is defined as 
a readiness or predisposition to act (Fishbein & Ajzen, 1975) such as the intent to use web resources in the 
classroom. The three components together provide an overall measure of an individual’s attitude. 

Attitude Instrument Face and Content Validity 

The original attitude survey created consisted of 42-item stems, 21 positively stated and 21 
negatively stated, written to represent the three constructs of attitude toward the use of web resources in the 
classroom. The instrument was processed through a series of validation procedures. Two separate expert 
review phases were conducted by educational psychologists, educational technology specialists, advanced 
graduate students in education, and K-1 2 practicing teachers to determine face and content validity as well 
as readability of each item stem. The second review, with a new set of experts, focused on interpretation of 
item stems and clarity of item stem language. Resulting data were analyzed and used to revise the survey 
items. 

The resulting 21 -item stems included 13 positively and 8 negatively stated items, ten belief, ten 
feeling, and one behavioral. The resulting item stems were randomly grouped in two sections on the survey. 
In the first section respondents were asked to indicate their level of agreement for each item using a scale 
ranging from strongly agree to strongly disagree. In the second section respondents were asked to indicate 
with which two semantically opposite terms they most closely identify with regard to using web resources 
in the classroom. All items were rated on 5-point likert scales scoring -2 to 2 points. The total attitude score 
could range from -42 to 42. The more positive the score the more positive the respondent’s attitude toward 
the use of web resources in the classroom. 

Construct Validity 

A total of 154 K-1 2 teachers, 43 from the United States and 1 1 1 from the Catholic Education 
System in Thailand, responded to the 21 attitude items. The teachers represented a mix of K-1 2 grade levels 
and various levels of teaching and web resources use experience. The teachers from the U.S. and Thailand 
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were participants in a workshop presenting ideas on how to integrate NASA web resources into their 
teaching. 

The survey was translated into Thai for the teachers from Thailand. It was then translated back 
into English, by two different translators, to ensure that the meaning of the questions were consistently 
interpreted by both groups. All teachers completed the survey prior to attending workshops on how to 
integrate web resources in the classroom. 

A principal components analysis of item inter-correlation followed by a Varimax with Kaiser 
Normalization rotation was carried out to generate factors hypothesized to reflect attitude constructs. 

Results 

A heterogeneous group of forty-three K-12 teachers from various locations in the United States 
included: eleven percent K-4th grade, forty-two percent 5th -8th grade, and forty-seven percent 9th -12th 
grade teachers. Sixteen percent reported having 2-5 years of teaching experience while the remaining 
eighty-four percent have more than 8 years of teaching experience. Twenty-three percent reported that they 
were novices or have never used the Internet, while sixty-eight percent reported they had intermediate skills 
and the remaining nine percent reported to be experts. (See Table 1 .) 

The 1 1 1 Thai teachers were made up of the following demographics: forty-one percent taught K- 
4th grades, twenty-eight percent taught 5th - 8th grades, and twenty- four percent taught 9th - 12th grade. 
The remaining seven percent indicated that they taught across the grade categories. In terms of teaching 
experience, eleven percent had taught less than 2 years, seventeen percent between 2 and 5 years, fourteen 
percent between 5 and 8 years, and fifty-eight percent taught more than 8 years. Fifty-six percent reported 
to be novices or never used the web in school, twenty-three percent reported to have intermediate-level 
Internet skills, and twenty-one percent reported to be experts. 

Table L Teacher Demographics 



Percent of 


Percent of 


U.S. Teachers 


Thai Teachers 



Grade level 



K-4 


11% 


41% 


5-8 


42% 


28% 


9-12 


47% 


24% 


across grade levels 


0% 


7% 


Teaching experience 






Less than 2 years 


0% 


11% 


2 to 5 years 


16% 


17% 


5 to 8 years 


0% 


14% 


more than 8 years 


84% 


58% 


Self-rated skill using 






Web 






Novice 


23% 


56% 


Intermediate 


68% 


23% 


Expert 


9% 


21% 


Note: U.S. teachers n=43; Thai 


teachers n=l 1 1 





The survey items loaded to three factors consistent with the hypothesized constructs of attitude, 
namely cognitive, affective, and behavioral. One of the cognitive (belief) stems loaded with the affective 
(feelings) components, one of the affective stems loaded with the cognitive components and another with 
the behavioral component. (See Table 2.) 
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The alpha internal consistency reliability score for all data collected was found to be .85. The 
alpha internal consistency reliability score for the nine factor- 1 items, loading on the feelings construct, was 
.76. The ten factor-2 items, loading on the beliefs construct, had an alpha internal consistency reliability 
score of .65. The two factor-3 items, loading on the behavioral construct, had an alpha internal consistency 
reliability score of .61. 

Table 2. Factor Loadin 2 for Teachers* Attitudes Toward the Use of Web Resources in the Classroom 



* 


Item 


Factor 1 


Factor 2 Factor 3 


F 


Using web resources in the classroom is gratifying to me 


.84 




F 


Using web resources in the classroom is urgent 


.78 




B 


Using web resources in the classroom is efficient 


.78 




F# 


Using web resources in the classroom is bad 


.74 




F# 


Using web resources in the classroom is foolish 


.74 




F 


Effectively incorporating web resources into lessons is essential. 


.72 




F 


Using web resources in the classroom is inviting 


.71 




F 


Using web resources in the classroom is non-threatening 


.66 




F# 


Using web resources in the classroom is boring 


.55 




B 


Effectively incorporating web resources into lessons can help 
present subject matter in realistic situations. 




.91 


B# 


Effectively incorporating web resources into lessons can 
decrease student interaction with instructional material. 




.88 


B# 


Web resources should be used only as “stand alone” instruction 
so that students can complete lessons on their own. 




.87 


B 


Effectively incorporating web resources into lessons can increase 
student collaboration during instruction. 




.74 


B 


Effectively incorporating web resources into lessons can enhance 
instruction. 




.72 


B 


Using web resources in lessons can have a positive impact on 
students’ attitudes toward learning. 




.72 


F 


Web resources can be effectively incorporated into existing 
lessons. 




.70 


B# 


Using web resources in lessons can have a negative impact on 
students’ achievement on tests. 




.54 


B 


The World Wide Web provides instructional resources that I 
have no other way of accessing. 




.45 


B# 


Using web resources in the classroom is inhibiting to students 




.43 


F# 


I do not want to use web resources in the classroom 




.68 



B I will use the web resources 

search engines or e-mail) in 


(such as informational web sites, 
my lessons. 




.40 


Mean attitude score: 


24.70 


Alpha reliability factor 1: 


.76 




Standard deviation: 


7.87 


Alpha reliability factor 2: 


.65 




Overall Alpha Reliability: 


.85 


Alpha reliability factor 3: 


.61 





# Item reflects negative stems that were reverse-scored 

* Code for hypothesized factor dimension: B = Belief, F = Feeling, U = Intended Behavior 



A post-hoc analysis was conducted to investigate data reliability for the two different samples of 
teachers. This resulted in an estimated alpha internal consistency score of .90 for the United States teachers 
and .65 for Thai teachers. 
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Discussion and Future Research 



The Teachers* Attitude toward the Use of Web Resources in the Classroom instrument was 
administered to a heterogeneous mix of K-12 teachers. The factorial analysis demonstrated that the items 
loaded consistently with the theoretical constructs of attitude, e.g., cognitive, affective, and behavioral. 
Further analysis of the overall internal reliability for the internal consistency of the item-stems revealed that 
the data collected with this instrument were highly reliable overall and for the U.S. teacher sample and 
moderately reliable for the Thai teachers. 

These data provided insights into the structuring of the item-stems to measure attitude. However, 
further analysis is needed to investigate the interpreted meanings in the semantic differential item stems. 
Questions were raised during a final review of the survey as to whether the opposite- word pair questions 
were indeed using terms that were opposite in meaning. For example, were the teachers interpreting 
efficient and time consuming as truly opposite in meaning? More work is required to ensure the proper use 
of terminology in the semantic differential item stems. 

Given the nature and number of resources available on the web there is also the possibility that the 
single question used to assess the behavior component of attitude was not robust enough to capture teacher 
intent and readiness to use web resources. There are literally thousands of types of web resource available 
to be used by teachers in a variety of ways. Intent to use one type of web resources in multiple ways or 
several types of web resources in the classroom may be lost using only one behavioral question, thus 
skewing the resulting attitude score. An equal number of behavioral stems need to be developed to match 
the number of cognitive and affective item stems. The additional behavioral item stems would provide a 
more complete measure of a teacher’s intent to use web resources in the classroom given the variety of web 
resource available for use in the classroom. 

Finally, the sample of teachers used for the initial testing of this survey was quite small. Once the 
instrument is revised based on the findings in this analysis it should be administered to a much larger 
population of teachers and new validity and reliability measures calculated. It would also be prudent to 
more closely investigate the differences between the U.S. and Thai teachers to understand why the large 
difference in reliability measures between the two groups. 

Conclusion 

Given the rapid growth of web resources available to teachers and students around the world, 
measuring attitudes towards the use of web resources will be critical in establishing appropriate 
interventions that help teachers integrate these new resources into the classroom. Teachers intent to use 
these resources begins with their attitude toward using them in their classrooms. This instrument, Teachers’ 
Attitude Toward the Use of Web Resources in the Classroom, has been shown to have face, content, and 
construct validity and produce highly reliable data. 

Although further enhancements are recommended to this instrument, data collected through this 
instrument can make a very important contribution to assessing the interventions used to help teachers in 
the process of integrating web resources into their classroom lessons. 
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Web-Enhanced Learning Environment Strategies: 
Integrating NASA Web Resources into Science 

Instruction 
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Penn State University and NASA Dryden Flight Research Center joined forces to investigate K-12 
teachers' use of web resources to enhance science instruction. The research resulted in the 
creation of Web-Enhanced Learning Environment Strategies (WELES). Several workshop sessions 
were held to determine how best to present WELES to teachers. The findings emphasize the need 
to provide contextualized practice, time for reflection, and follow-up to support teacher use of 
these new NASA resources. 



Introduction 

“Teacher technology training tends to take the form of an introduction to the mechanics 
of the equipment, rather than how to incorporate it to do teaching jobs more effectively. 
There is seldom incentive to devote the energy needed to use technologies innovatively ... 
What exists generally focuses on teaching about the technology rather than teaching with 
technology . . . they need training to create classrooms that allow students to achieve new 
standards.” (Hawkins, 1998) 



One of the key responsibilities of a teacher is to plan lessons that are taught with effective methods 
of teaching using the best resources available. Teachers are especially good at using effective methods of 
teaching, but find it especially challenging to locate, purchase, beg, or borrow valuable teaching materials. 
Once found, the material is often deemed out-of-date, inaccurate or irrelevant to the day’s lesson. 

As a remedy to the problem of limited access to good teaching material, the World Wide Web 
(WWW) has been billed as a substantial resource for current, relevant, and easy-to-access material. NASA 
has invested heavily in creating a vast array of web resources that are available to teachers. In November 
1 998, 29 1 ,470 web pages were found with the nasa.gov domain name. This number, of course, does not 
take into consideration contracted work. Often, however, being a vast resource, the WWW is also 
perceived as being an overwhelming wilderness that can intimidate even the stout hearted. Others believe 
that to use this resource, one must invest a substantial amount of time learning to publish homepages and 
lesson plans on it. These beliefs need to change. 

The Pennsylvania State University and NASA Dryden Flight Research Center joined forces to 
investigate how web resources, specifically those produced by NASA, could be used by teachers to 
enhance instruction and inspire students to pursue science, math, and technology fields. The three-year 
project focused on trying to understand the activities teacher perform to plan, create, and manage classroom 
iiistructioii and the various types of resources available llu ough web technology and use these findings to 
create interventions that help teachers enhance science instruction with NASA resources. The results of this 
research are the Web-Enhanced Learning Environment Strategies (WELES) Reflection Tool and Lesson 
Planner. 
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Web-Enhanced Learning Environment Strategies - WELES 



WELES is a roadmap for guiding teachers through the specific categories of resources available 
on the Web, and a framework for reflecting about how these different types of resources may be used with 
six methods of teaching using four key lesson strategies. The resulting four central Web-Enhanced 
Learning Environment Strategies (WELES) follow a typical lesson sequence: framing a lesson, informing 
learners, providing for content exploration by the learner, and allowing learners to try out newly acquired 
skills, knowledge, and inclinations. 

The Reflection Tool is provided as an overview of the conceptual interrelationship between 
teaching methodology and World Wide Web categories. Teachers can often see from this overview, a 
broader conception of how the web can be used for classroom teaching. See Figure 1 . 
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Figure L WELES Reflection Tool 

The Lesson Planner, on the other hand, was developed as a practical tool for teachers to use when 
actually planning a specific lesson. The planner takes the teachers through the thinking process step by 
step to be sure that they think about all of the teaching options that they have in using this vast resource. 
See Figures 2 and 3. 



164 



BESTCOPY AVAILABLE 



169 



z. 

z 

< 

PLh 

Z 

o 

Ifl 

Ifl 

Xfl 



pfi 

0^ 



< 

C/D 



C 

0 

T3 

0> 

u 

fi 

c« 

JS 

fi 

1 

pfi 

a> 



bi 

a> 

cl 

C/D 

bi 

fi 



u 

cl 

a> 

H 









DO 



pp-Wj 

3> 



VO 



o 

ERIC 



Figure 2. Page 1 of WELES Lesson Planner 








Figure 3 . Pa^e 2 ofWELES Lesson Planner 



The resulting design took into consideration our beliefs about teaching and web resources, our 
understanding of changing pedagogy, changing realities of classroom access, change research, and 
extensive feedback from teachers. We held the following four beliefs and assumptions related to 
contemporary methods of teaching with WWW resources. 

• The first belief is that by taking a broad perspective of the WWW and the types 
of resources it contains, it will become more useful to the teacher. 

• The second belief is that WWW resources should be used to enhance lessons 
more often than it is used as a public space for publishing homepages or teacher 
developed lessons. 

• The third belief and assumption is that teachers are already teaching using 
contemporary and effective methods of teaching in a manner that engage 
learners with as many resources as they can gather together, but are challenged 
by the latter. 

• Fourth, and finally, we believe that WWW resources can be useful for one or 
more lesson strategies, or for the lesson as a whole. 

To reach the greatest number of teachers, we felt that the model as we designed it needed to 
include both traditional and contemporary teaching methods. Traditional teaching methods included the 
Present, Guide, and Active Learning methods in which the teacher is in more direct control, reflective of a 
behavioristic and cognitive pedagogy. We then included role playing, problem-based and collaborative 
learning in which the teacher plays less of a direct role, as reflective of a more constructivistic pedagogy. 

While not represented in the figure, WELES also provides advice to teachers that takes into 
consideration 6 different computer/Intemet access configurations. We emphasize that if the teacher only 
has access at home, or in an administrative area, there also are ways for him or her to use this resource. 
Primarily, the teacher uses the resource as a planning tool for a vast number of lesson plans and resources 
that he or she can print out and bring into the classroom. For those who have lab access, one computer with 
or without a display panel, multiple configurations directly in the classroom, the options for Internet 
integration grow. 

Theories on teacher adoption of technology innovations greatly influenced the design of WELES. 
Research on teachers suggests that those teachers at ease with technology quickly migrated towards more 
sophisticated and challenging lessons incorporating new technology-enhanced resources and methods 
(Geyer, 1997). Encouraging teachers to reflect on lessons, activities, and strategies they were familiar with 
and that have worked well in the past helped them adopt new technologies (Schdn, 1983; Canning, 1991). 
By using an existing repertoire of successful assignments, projects, and activities from their tool kit, 
teachers had enough familiarity with old procedures to ensure a successful instructional outcome when 
applying new technologies. Continuing to experiment with web technologies in steps encouraged teachers 
to include web resources as a regular and continuing part of instruction rather than as rare, anecdotal 
experiences. Yet, the first step is to encourage teachers to think about successful instruction, current 
resources, and how technology can be comfortably integrated into those successful lessons. This process of 
reflection allows for the building of logic and movement toward change (Senger, 1998). WELES was 
designed to introduce web technology innovations through methods and terms with which the teachers 
were already familiar, and had experienced success. It was important that the model guided teachers slowly 
since they are more likely to incorporate innovations a little at a time until they become comfortable with 
the innovation. 

Lastly and most importantly, the reflection tool and lesson planner were forged and shaped by 
teacher feedback over a 3-year period. We offered 8 NASA-sponsored workshops attended by 164 teachers 
in the United States, 5 Penn State University graduate courses attended by 93 teachers, and 1 international 
workshop attended by 118 Thai teachers. After each session, we considered all of the feedback, and revised 
the model accordingly. 

Our next step was to determine the best way to diffuse the WELES reflection tool and lesson 
planner. Results and the methods of delivery for each workshop were analyzed to determine if the WELES 
was not only an effective organizer, but if teachers actually integrated NASA resources into their science 
lessons in the long run. 
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WELES Workshop Description 



One of the WELES workshops was presented within the context of a NASA teacher ambassador 
program. The teachers were immersed in NASA content over a two-week period, learning about many 
different types of NASA activities and resources that they could take back to their classrooms. During two 
of the days the teachers were focused on learning about and using WELES to create lessons using NASA 
web resources. Following this workshop, teachers participated in an 8-month 3-6 variable credit graduate 
distance education course requiring further discussions and development of WELES lessons incorporating 
NASA web resources. 

The second type of workshop focused solely on describing WELES and providing teachers with 
practice in developing WELES lessons. The examples presented and sites visited during the workshop were 
primarily NASA-related sites. The workshop follow-up required the teachers to complete daily logs 
recording their use of web resources over an 8-week period. 

Long-Term Impact 

Approximately one-year after these sessions, long-term follow-up surveys were administered to 
assess NASA web resource use. More specifically, we were interested in determining the frequency in 
which teachers integrated NASA web resources, the types of school subjects in which they were used, the 
types of NASA web resources (e.g. images, interactive events, scientists), and the reasons teachers stated 
about why they used them. We were also interested in comparing the patterns of results from the teachers 
who attended the two different types of WELES workshops. 

Ninety-four of the original 164 teachers who participated in the WELES workshops provided an e- 
mail address for future communications. Each was sent a follow-up e-mail survey gathering responses to 
six questions regarding their use of NASA web resources. Three of the e-mails were returned due to 
delivery errors. Approximately fourteen percent of the teachers responded to the questions in the requested 
time frame. 

Eight of the thirteen teachers responded that they did not know about NASA’s web resources prior 
to attending the WELES workshop. Five responded that they did know that NASA had web resources, four 
had indicated that they had used NASA web resources in their classrooms. This pattern continues to be 
consistent with workshops that we continue to conduct. 

Nine teachers indicated that they used NASA web sites in science classes, two used them in math 
classes, four in technology classes, two in a geography class, and 1 in an aviation class. One indicated that 
she was on sabbatical but planned on using NASA resources when she returned to teaching. Some teachers 
use NASA resources to teach multiple subjects. Two of the teachers, not on sabbatical, did not use NASA 
web sites at all. 

With regard to the NASA sites they were using, most participants provided general descriptions of 
the sites they have used to teach. These included: nasa.gov, SR71, weather, space, atmosphere, airplane 
ground school, airplane construction, geology tracking, JPL, planets, NASA control panel, how machines 
work, Galileo, Mars Pathfinder and links. Some also indicated that they have used too many to list or many 
others. 

When asked about which types of web resources they were using, ten indicated that they used 
images, nine used number or picture databases, six used NASA lesson plans, nine used general NASA 
information sites, and two have communicated with NASA people. Two have been involved with NASA 
interactive student project sites, one has used NASA tutorials, and no one indicated using any other types of 
NASA web resources. 

When asked about what they liked best about NASA web resources, teachers gave eight positive 
responses. They indicated that they thought that the NASA web resources increased student interest, 
provided information not easily obtained otherwise, provided beautiful graphics, were interesting, provided 
cutting edge information, were vast and informative, contained current science events, and were useful for 
teaching. One teacher stated that she was able to find information that all of her students (grades 2-6) could 
understand and that they really enjoyed the interactive sites. One also mentioned that she most liked 
bringing scientists and experts into the classroom through the NASA web sites. 




169 

175 



Teachers indicated that to improve NASA web resources, they would like descriptions of specific 
sites with specific addresses to get to the specific information or that there should be an easily accessible 
catalog of NASA web resources. 

The teachers in the contextualized workshops reported using NASA web resources more 
frequently and using more types of resources in more topics. There were also some interesting patterns in 
the types of resources used. For example many more teachers from the contextualized workshop were using 
the information-based resources, e.g. images, animation, and lesson plans than people resources, e.g. 

NASA scientists and experts. These teachers also used a variety of NASA sites as well as other science web 
resources. Teachers commented that they have used teacher strategies with these web resources that they 
have not regularly used in the past. For example a teacher who primarily used present strategies began to 
use more problem-based learning and collaborative learning methods in her class. Another teacher who was 
primarily using active learning methods admitted to using more present methods because of the interesting 
materials on the web and lack of access for all students. 

Conclusions 

The results of our three year effort leads us to conclude that the WELES lesson planner and 
reflection tool makes an important contribution to helping teachers think about the web and how it can be 
integrated into their classroom teaching. An important conclusion, however, regarding diffusion of the 
model, is that the more contextualized the workshop describing WELES, the more likely the Web will be 
used to teach specific topics. If one’s goal is to enhance science teaching with NASA resources, introduce 
WELES in an existing NASA teacher workshop. If one’s goal is to enhance teaching in general without 
regard to resources, methodology, or school subject, then a more generic workshop would be appropriate. 

A second very important conclusion, is that follow-up is important, but not in the form of a listserv. The 
workshop that extended over the 8-month period of time had a greater impact on teacher use of the web, 
than those who attended a short-term continuous session. 

These conclusions are to be interpreted cautiously as the long-term impact data only represent a 
small percentage of the individuals who actually participated in all forms of the WELES workshops. It will 
be very important to follow-up with the remaining participants to determine how and if they are using 
NASA web resources in their classroom. 
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The lack of professional benefits and the barriers to such an approach for new researchers in 
academic tracks have made collaborative research methodologies and co-authoring less 
appealing than single-authorship. However, collaborative research can be used to enhance the 
teaching and learning of new and experienced scholars. Following a discussion on the advantages 
and disadvantages of collaborative research, a suggested model for conducting collaborative 
research is presented and benefits and challenges to using collaborative research methods for 
dissertations are discussed. 



Introduction 



“There is a clear presumption in research oriented communication and 
social science literature that research is and should be an individual 
effort . . . multi-method researchers have criticized the general 
disregard for collaborative work and have argued that devaluing it 
harms both researchers and the knowledge that it generates.” (Violanti, 

1992, p.65) 

Webster's Dictionary (1992) defines collaboration as working together or jointly. Research 
scientists more commonly define collaborative research as a process where two or more researchers or 
investigators work jointly on a project and contribute resources and effort, both intellectually and 
physically ... offering suggestions, collecting and analyzing data, writing, and editing (Rekrut, 1997; 
Violanti, 1992). The degrees of collaboration range from full collaboration where all collaborators are 
involved in all phases of the research and share authorship to partial collaboration where investigators 
contribute to different phases of the research process and choose to publish individually. 

Collaboration, while not new, has not been widely recognized nor rewarded in academe. Incentive 
systems including tenure, promotion, and salary for researchers and scholars generally prioritize and 
compensate based on the traditional individual research determined in part by publication history (Violanti, 
1992). Most research projects however, are indeed collaborative efforts. When scholars borrow evidence 
from existing research, base research on unanswered questions in the literature, or ask others for assistance, 
they engage in collaboration, typically without recognizing or acknowledging that as part of a collaborative 
research process. The culture of academe, as exemplified in tenure discussions and decisions, seems to 
devalue collaborative efforts acknowledged through joint authorship by valuing single-authored works 
more. This value system raises the question as to whether collaborative research should be promoted. 

Review of the Literature 

The literature provides at least two perspectives on the acceptance of collaborative research as a 
scholarly practice. One is directly evident by quantifying the number of articles published by single and 
multiple authors in highly regarded scholarly journals. The other is illustrated in the descriptions, 
discussions, and instruction provided on research methodology. 
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Empirical Patterns of Authorship 

The value of collaborative research can be deduced from empirical patterns of authorship in 
scholarly publications. The authors do recognize, however, that various scenarios can account for the ratio 
of sole to multiple authored published works. For example, the authorship patterns may represent editors’ 
perceived value of single- versus multiple-authored works; collaborator’s may not have been identified in 
the authorship of manuscripts; or authors may have agreed to publish separate articles individually for 
tenure, promotion, or vitae purposes (Heffher, 1981; Violanti, 1992), However, it may also be likely that 
the pool of submitted articles is more greatly weighted in one direction or another indicating professor 
behavior consistent with what is valued for tenure. Thus, studies on empirical patterns of authorship should 
be expanded to clarify the reasons behind the patterns of single and multiple authorship, 

Violanti’s (1992) review of authorship patterns of seven mainstream social science and 
communication journals published between 1980 and 1 989 revealed that upwards of 80% of all articles 
were single-authored. In that time period the average percent of single-authored articles for each of the ten 
years ranged from 47% to 80%. Although no firm generalizations were drawn from these results to other 
disciplines, the seven journals examined were quite similar and representative of the social sciences, 

A similar review of highly acknowledged instructional design and educational research journals, 
showed slightly different patterns from Violanti. Of the nearly 1400 articles published in Educational 
Technology Research and Development, Tech Trends, Educational Researcher, Journal of Instructional 
Science, and International Journal of Instructional Media, between January 1990 and August 1998, there 
were significantly more single-authored articles overall, f (1,43) = 6.68, p>,013 (mean = 19 single-authored 
articles per month, mean = 1 2 multi -authored articles per month). Over 61% were single-authored, (See 
Table 1.) 

Table /. Authorship patterns between January 1990 and Ausust 1998 



Journals (Associations) 


Total 

Articles 


Single 

Author 


Multiple 

Authors 


Percent 

single 


Percent 

multiple 


Educational Researcher (AERA) 
International Journal of Instructional 


293 


228 


65 


77.8% 


22.2% 


Media 


418 


309 


109 


73.9% 


26.1% 


Tech Trends (AECT) 


291 


162 


129 


55.7% 


44.3% 


Journal of Instructional Science 


161 


75 


86 


46.6% 


53.4% 


ETR&D (AECT) 


226 


80 


146 


35.4% 


64.6% 


Grand totals: 


1389 


854 


535 


61.5% 


38.5% 



Educational Researcher typically acknowledged as a very traditional and conservative journal and 
the International Journal of Instructional Media had the highest percentage of single authored articles 
(77.8% and 73.9% respectively) over an eight- year period. ETR&D contained the highest percentage 
(64.6%) of multiple-authored works. The Journal of Instructional Science and Tech Trends had a fairly 
even mix of single- and multiple-authorship of articles between 1990 and 1998. 

Although there were more single-authored publications overall in the instructional design and 
educational research journals, the pattern shows some important shifts in some journals toward multiple 
authored works- especially for one of the most important mainstream research journal in the instructional 
systems field. This very speculative analysis may suggest that there is reason to believe that more 
collaborative research will be acceptable to at the instructional systems academic environment and that 
collaborative research strategies should be pursued as a scholarly activity. 

Defined Research Methodology on Collaborative Research as Scholarly Practice 

Violanti (1992) also conducted a content review of the communications and social scientific 
research literature, including many of the same texts used to guide educational research. She found very 
little mention of collaboration in research textbooks and no mention of co-authorship patterns. In fact, she 
found that most textbooks that do mention collaborative research often focus on the negative aspects of co- 
authorship such as smaller royalties, extra accounting, author hand-holding, stress, and resentment. When 
textbooks did list advantages of collaboration and co-authorship they generally mentioned endorsement of 
an expert, credibility, multiple views of the research issue, and someone else to send on the promotional 
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tour. Although these aspects of collaboration may be important to those involved, they neglect to 
emphasize several key advantages of conducting collaborative research. With so little discussion on 
collaborative research methodology, one could also speculate that high single authorship patterns are the 
result of a lack of skill in collaborating on research projects or a lack of experiencing its genuine value. 



The Value of Collaborative Research 

“Why collaborate? ... to efficiently use time and labor, gain 
experience, train researchers, sponsor a protege, increase productivity, 
avoid competition, multiply profeciencies, access special equipment, 
skills, visibility, or recognition (Violanti, 1992, p. 67). . . mutual 
stretching and sharing allows achievement of a vision richer than any 
individual could achieve alone. . . .multiple perspectives . . . support of 
each collaborator (p. 69).” 

The current literature on collaborative research, although scarce, suggests multiple benefits of 
conducting collaborative research. Garside (1996) found that working collaboratively gave participants a 
chance to learn new information from other researchers, consolidate their own learning and integrate it with 
other concepts, sharpen their own opinions, and clarify their own points of view. Collaborative efforts 
promoted critical thinking because participants interacted with material through discussions with others, 
viewing ideas and findings from multiple perspectives, and identifying and remedying errors of individual 
judgement. In addition, Lieberman (1986) found that successful interactive collaborative research yielded 
several important benefits. Benefits of collaborative research included increased skills in collaborative 
decision making, data collection, and material development, early problem identification emerging from the 
team’s mutual concerns and inquires, increased team attendance to both research and development 
concerns, and increased reflection and self-learning during the research process. Collaborative research 
enhanced the evaluation process and sharpened the conceptualization of the goals and protocols for those 
who served as research leaders (Galinsky, Turnbull, Meglin, & Wilner, 1993). Also, team development 
grew out of a continuous need to deal with problems and reaffirm the teams’ understanding of the research 
(Galinsky et al., 1993). 

Collaborative research, however, has its challenges. Collaborative research generally takes more 
time than individual research efforts and needs to be carefully planned and managed. Decision-making 
must be flexible to accommodate multiple agendas and perspectives. Individual researchers can become 
frustrated trying to rationalize and meet their own research agendas while balancing the needs of other 
researchers. A researcher must weigh the benefits against the challenges to decide between the two 
strategies. 

“Why choose to collaborate? ... interest in working with others to gain 
expertise and knowledge others have as well as to share one's own 
knowledge and expertise with others . . . take advantage of the 
collaborative process, multiple perspectives, share and critique of ideas, 
formation of new and expanded social and scholarly networks, quality 
checks on work, socialization of new scholars into the research process 
and improving intra-departmental and inter-departmental relations.” 

(Violanti, 1992,p.79) 

The interpretation of research results may be enriched by collaborative research efforts by 
capturing data from multiple perspectives and tapping into the different resources and strengths of each 
investigator (Menon & Owens, 1994; Rekrut, 1997; Violanti, 1992). In addition, a mentoring relationship 
between experts and novices in terms of both the research process and subject matter perspectives may 
develop, thereby improving the quality of current and future research (Menon & Owens, 1 994; Rekrut, 
1997; Violanti, 1992). Menon & Owens (1994) found that collaborative research projects fostered 
professional development and increased the participants’ confidence in teaching and researching. The 
experience in collaborative research encouraged participation in similar projects and sharing experiences 
and learning among colleagues. Novice researchers reported that they learned how to study problems more 
systematically. Collaborative research teams reported that working together enhanced their understanding 
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of themselves and others (Rekrut, 1997), both important characteristics of a researcher. Thus, collaborative 
research provides value to both the novice and expert in that the process of collaboration enhances the 
research skills, abilities, and understanding of the investigators. The potential implications of successful 
collaborations of this nature include increases in learning about the process of research (Galinsky et al., 
1993; Lieberman, 1986), increases in collegial interactions and greater knowledge of the research area 
(Galinsky et al., 1993), more sound research products, and demonstration of the individual’s research 
knowledge and skills. 

A Model for Collaborative Research 

Given the benefits of collaborative research, and the need to fill a void in descriptive research 
methodology, the authors created a model of collaborative research which tracked an actual collaborative 
research project. This model follows generally accepted standard research methodological steps and 
identifies specific activities when the individual researcher needs to focus on his/her agenda and 
recommended activities for team collaboration. (See Figure 1.) 

Establishing a Team 

Establishing a workable team by choosing colleagues with, first, similar interests and, second, a 
willingness and understanding about how to work together is critical. Thus, prior to beginning any 
collaborative research effort, members must be identified who represent a variety of interests that have 
some common link. Each member may indeed focus on individual questions of interest, but the 
commonalties that cross all projects may make a collaborative environment beneficial. While willingness 
to work together is necessary, it is not sufficient for a successful research effort. Planfully addressing the 
commitments required for successful collaboration will help identify if the team is ready and willing to 
collaborate. As suggested by Rekrut (1997) an important part of establishing any collaborative research 
team is developing an understanding of the dynamics of the group and providing guidance to members to 
ensure successful outcomes. Coordination is essential. 

Questions that should be answered when establishing a collaborative research team include: 

• what problems are each of the individual researchers interested in investigating, 

• what theoretical foundations and research methodologies and frameworks are 
important to each researcher, 

• how can the individuals work as a team, and 

• what are the strengths and weaknesses of each player? 

Collaborative research requires careful planning and execution to result in sound investigations 
and conclusions. Following generally accepted research protocol requires defining the research questions, 
developing a research proposal that includes descriptions of variables, subjects and participants, research 
methodologies, treatments, instruments, protocols, and analysis techniques, and conducting the research 
and reporting results (Krathwohl, 1993). In a collaborative research project, certain of these activities 
should to be defined and conducted by an individual researcher based on his/her research interests and 
questions. Other tasks require discussion and coordination among participating researchers. It is important 
to continually check variables, subject, methods, protocols, and treatments as to not contaminate the results 
from other researchers’ inquiries throughout the collaborative research process. 

Defining Research Questions and Conducting the Literature Review 

After establishing a research team one of the most challenging aspects of collaborative research is 
to Clearly define and operationalize the research questions. In this model of collaborative research, the 
collaborative research team works together to define the overall research goal while refining each 
member’s distinctive research questions. At this point, it is important not to compromise initial individual 
curiosity or the integrity of each other’s research inquiries. 
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Figure /. Collaborative Research Process 

Working together on defining the research questions, supported by individual literature reviews, 
should focus on the prompting clarification, definition, and refinement of individual and group research 
questions. The collaborative process can provide each researcher with a sounding board to further explore 
aspects of questions that may not have necessarily been considered. Collaboration during this phase of the 
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research may take the form of a series of discussion sessions, meetings between senior and novice 
researchers, and sharing of initial concept papers based literature reviews. 

Questions considered by the team at this point should include: 

• what are the research questions being addressed, 

• what are the hypotheses, 

• what variables are being measured, 

• under what conditions do data need to be collected, and 

• who are the subjects of the research? 

Although the ultimate goal of this initial step is to define research questions, the process of 
refining the research questions will generally continue well into the next several steps of the research 
process. 

Developing a Research Proposal: Designing a Collaborative Research Methodology 

The next key step in this model is to develop the research proposal. Th research proposal generally 
includes a statement of the research problems and questions, hypotheses, definition of study variables, 
subjects, treatments, research protocols and methods, and analysis techniques (Gall, Borg & Gall, 1996). 
Defining the subjects who will participate in the research and research treatments requires collaboration 
among all team members. For example, if the questions are focused on the impact of new curricular 
materials on high school teachers and students, the team works together to develop subject participation 
criteria, the new curricular materials and classroom administration procedures, and identify sources for 
acquiring high school subjects. 

A critical part of this step is to make sure that the treatments and implementation protocols 
provide the research environment of interest to each researcher. For example, where one researcher may be 
interested in differences before and after treatments another may be interested in the presence and absence 
of the treatment, and still another interested in different applications of the treatment. Each can gather data 
from the participating subjects using some of the same treatment environments but each may also require 
different treatment groups. 

This model suggests, however, that defining variables and research methods are initially the 
responsibility of the individual researchers based on their research questions. Since individual research 
questions will most likely be investigated through different means, the variables of measure need to be 
defined for each research question. The methods used to collect data, either quantitative or qualitative, will 
also directly result from the research question and identification of variables. Thus, each researcher should 
develop definitions of the variables and methods to collect their required data. Once defined on an 
individual level, the researchers should meet to discuss the variables and research methods to assure that 
data requirements and research methods are aligned, not overly excessive for the subjects, and not in 
danger of contaminating each others’ work. 

Team meetings focusing on describing variables and research methods and sharing of research 
proposal development documents are ideal methods for collaborating on the research proposal. Questions 
addressed by the team at this point should include: 

• what are the independent and dependent variables of interest, 

• what instruments and procedures are used to collected data, 

• what treatments are required, 

• what research methods need to be employed, and 

o how might the research methodologies interfere or support each research 
question? 

It is also critical to re-examine the research questions during this stage to make sure that the 
variables and methods do indeed provide a framework that will result in appropriate data. All factors of 
each individual’s work as well as the goals of the research as a whole should be agreed upon in the final 
version of the research proposal. 
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Conducting the Research 

The main goal in this stage is to administer the research treatment(s) and gather data in support of 
research hypotheses. There are several challenges in this phase of collaborative research. The activities of 
conducting collaborative research test the abilities of the team to work effectively in a collaborative 
relationship. Too often it is easier to conduct the research focusing one’s our own agenda, however, a 
focus on the individual will often compromise the quality of the other researchers’ work. Maintaining high 
ethical standards, continually reviewing the research questions and treatment descriptions, and maintaining 
a high level of communication is critical to successfully working though the challenges and ultimately 
completing the study securing quality data for each person’s investigation. 

Some of the most challenging aspects of conducting collaborative research can be coordinating the 
administration of treatments and data collection instruments and procedures. Reviewing treatment 
documentation and managing communications with team members and subjects is necessary to ensure that 
information will not be divulged that would contaminate each other’s research. Coordinating overall 
management of each stage of the research including contact with the subjects, administration of surveys, 
and data collection is a challenge that should not be overlooked in the planning process. On-going attention 
to the details should be a shared responsibility that incorporates continual communication among 
collaborators. 

Methods that may support collaboration when conducting the research include sharing electronic 
journals of activities, posting of treatment and data collection schedules, group reviewing of 
communication protocols used with subjects, and the entire team participating in issues forums to discuss 
problems and potential solutions. Questions that should be addressed collaboratively during this stage of 
collaborative research include: 

• when are treatments being administered, 

• who is involved in the treatment and data collection, 

• who is communicating with the subjects, 

• what problems are occurring with the research, and 

• how are issues being resolved? 

Analyzing and Reporting 

The goal of the analyze and report phase of this research is to explore the collected data and report 
on the results, generally by writing research articles. During the analyzing and reporting phases of 
collaborative research there are opportunities to work individually and collaboratively. Discussing the 
results of analysis with peers can provide individual researchers with rational for findings, explanations for 
specific results, and support for rejecting competing hypotheses. For example, in a recent collaborative 
research project the researchers investigated the effect of conversation on attitudes for treatment and non- 
treatment groups (Koszalka, 1997) and the effect of group size on richness of conversation (Toci, 1997). 
Non-significant differences in attitudes between different sizes of treatments groups were found. The non- 
significant results were explained by examining the results of the investigation of richness of conversations. 
The findings suggested that the conversations held with groups of different sizes were of the same level of 
richness (Koszalka, 1997; Toci, 1997). 

Collaboration on interpreting data results can promote richer and deeper understanding of 
respective findings. More importantly these types of collaboration can promote critical thinking, provide 
multiple perspectives on the interpretation and explanation of findings, and deepen the understanding of the 
research area and process. Whether researchers decide to publish separately for tenure and promotion 
reasons or to focus on a specific point of the research, there are opportunities to publish collaboratively to 
reach audiences with multiple perspectives, provide richer descriptions of individual research findings, and 
to share the accomplishments and learnings of the group. 

Meeting to plan publication authorship, responsibilities, schedules, and targeted journals and 
sharing and reviewing manuscripts is important in this phase of collaborative research. Questions that 
should be addressed collaboratively include: 

• what were the significant and non-significant findings from the data, 

• how do the findings relate to other researcher’s findings, 

• what explanations can be used to support findings. 
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• what competing hypothesis need to be addressed and can they be refuted with 
other findings in this research, 

• what articles and books should be written, 

• what schedule should be followed, and 

• who will take the lead on authorship? 

The Collaborative Research Model 

This model provides a visual representation of the possibilities for coordinating collaborative 
research. It identifies areas of focus for the individual as well as points of collaboration for those involved 
in the research. Such a model provides a framework for incorporating collaborative research as a teaching 
and learning tool for new scholars. This model provides an ideal environment for students to demonstrate 
their knowledge and skills during the dissertation process as well as allow supporting committee members 
the opportunity to observe new scholars and participate with them during the continuing learning process. 

Are Collaborative Dissertations a Viable Option? 

“The purpose of the doctoral dissertation is to develop and to refine 
scholarly skills in research and writing as well as to make a 
contribution to the literature in specific fields. An additional objective 
of a dissertation is to "hook” you on the idea of doing research and 
continuing this academic effort after you have completed the doctoral 
requirements . . . another purpose of the dissertation project might be to 
develop candidates' skills in working cooperatively and effectively with 
peers while also attaining the other university objectives.” (Thayer & 

Peterson, 1976, p. 126) 

Most professors in academe would agree that the purposes of dissertation include demonstrating 
the individual's competencies as a researcher, knowledge of a specific area of focus, and potential abilities 
make to contributions to the existing research base. However, the dissertation process varies in required 
processes and deliverables. Doctoral candidates are often required to have multiple committee members 
representing multiple disciplines related to their research topics and follow rigorous guidelines to plan, 
conduct, and report on research. In essence, a single candidate is collaborating with his/her committee 
members during the dissertation process. 

Collaborative research among many doctoral candidates may also provide unique benefits to the 
dissertation process. Once a research topic with collaborative potential is established, the candidates could 
benefit from exposure to multiple committee members representing each student’s field of study. 

Individual candidates would still need to write a proposal clearly articulating the research problem 
and questions, review of the literature, and conduct a separate component of the larger research project, 
while also collaborating with fellow candidates and advisors to effectively connect the research projects 
together. Collaborative research would include demonstrating the knowledge and skill generally expected 
in individual dissertation processes as well as skills necessary to conduct collaborative research. Each 
student would be responsible for coordinating of all phases of his or her own research as well as attending 
to the needs of other participating candidates and their advisors, specifically enriching their abilities in 
decision making and research planning. The ultimate product of the collaboration could be an individual 
dissertation, as normal practice in most dissertation process, with an added component reflecting on the 
collaborative research and learnings gained during the process. 

Acceptance and coordination of collaborative dissertations may be one of the greatest challenges 
to implementing such a process. A critical ingredient for acceptance is the program chair and facult)' buy-in 
to the concept of collaborative dissertation. Dissertation committee chairs, advisors, and members would 
also have to agree to the process, group of participating candidates, research agenda and procedures, and 
schedule. Coordination of multiple separate committees working together would be key in maintaining 
schedules. On the candidate side it will be critical to determine common interests and individual agendas as 
well as ensure that the students are willing to and capable of working together and being supportive of each 
others’ research. 
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The small group collaborative approach to dissertations would still include demonstration of 
individual competencies including demonstrating understanding of the topics being researched by the 
candidates successfully integrating their own research components into a group proposal. This would 
include integrating appropriate definition of the problems being investigated, literature reviews, protocols, 
and other key research information. Candidates would also have to demonstrate research project 
management abilities such as establishing protocols, working with subjects, and managing research issues. 
They would also still have to demonstrate their abilities to collect data, including developing and validating 
of measurement instruments, conducting observations, and recording data. Each would have to 
appropriately analyze their respective data and demonstrate understanding of the results of their research as 
well as the collective research of the group. Each would also be demonstrating his/her own writing skills by 
writing separate dissertations and acknowledging contributions of others and how they affected his/her own 
work. 

What would be the advantage of what seems to be a lot of extra work? The collaborative 
dissertation approach provides candidates with a group of peers who are going through a similar process, 
attacking similar questions from slightly different perspectives, and who will most likely have a variety of 
strengths and weaknesses that can benefit and support each other. There will be a potential for each 
candidate to develop a deeper understanding of the research topic area and demonstrate skills required to be 
successful in the research field. Candidates, as well as the advisors, will also have an opportunity to learn 
more about team research (Gall, Borg & Gall, 1996) and become better equipped to function on large 
research projects with multiple participants. 

There are, however, potential drawbacks to collaborative dissertation research. Generally, it is 
expected that more time and energy are required to work in groups and coordinate agendas. Individual 
goals may also be occasionally compromised and have to be re-conceptualized. There are also personal 
issues that reduce the effectiveness of collaborative dissertation, such as the work rate of some team 
members may be too fast or too slow causing others to be frustrated. A critical drawback on the advising 
side may be wide differences in other committee member's views of the research process or topic-making 
working relationships difficult. All of these potential hindrances need to be considered before beginning 
collaborative research. Considering the benefits and challenges of collaborative dissertations in conjunction 
with a collaborate research model such as the one proposed in this article can help support the decision 
process to proceed with such a dissertation. 

Conclusions 

As suggested in previous literature the acceptance of collaborative research, specifically as 
quantified in multi-authored versus single-authored articles, is not as valued as single research efforts. 
However the reasons behind single-authorship leave questions as to whether collaborative efforts have not 
been acknowledged so that the authors can gain the “credit" they need to enhance their academic careers. 
Literature on participants in collaborative research projects suggests that the collaborative nature of 
collaborative research provides multiple benefits to the individual. Collaborative research has been found to 
be a valuable learning experience for both the novice and expert researcher. It has been shown to promote 
critical thinking, enhance the comfort of novice researchers, and provide ample opportunity to demonstrate 
research process, topic knowledge, and abilities. 

The model presented here provides a suggested approach to collaborative research, outlining key 
tasks that should be completed by the individual researchers and where collaboration would provide benefit 
to the research as a whole. Although many permutations are possible given the scope of each researchers’ 
role, the discussions on the commonalties among the topics and procedures provide team members 
opportunities to Icam from one another, think critically about their own and their partner’s research, and 
develop heightened skills in argumentation and research process skills. 

A suggested application of collaborative research is in the dissertation process. Collaborative 
research promotes individual and team development and allows ample opportunities to demonstrate 
knowledge, skills, and abilities as a researcher. The model presented provides a more integrated approach 
to collaborative dissertation over the often noted process of doctoral candidates carving a piece of a larger 
funded research project (Gall, Borg & Gall, 1996). Candidates become a much more active part of a team 
of junior researchers, work together to define and explore questions, and demonstrate their skills 
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throughout the process. Such activities promote learning and further preparation for participating in the 
larger practice of educational researchers. 
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Abstract 

A qualitative analysis of nine college students using different types of scaffolding (instructional 
supports) while working on a World Wide Web project was conducted. Four different support 
mechanisms were examined: (a) WWW resources; (b) procedural guidelines for the instructional 
activity; (c) student-student interactions; and (d) instructor-student interactions. Two major 
findings are discussed: (a) the utility of scaffolds and WWW resources seem dependent on the 
ability of learners to readily grasp how the scaffold or resource could help them; and (b) social 
scaffolding based on face to face dialogue with instructors and peers was critical to helping 
learners manage the complexity of the open-ended project. Implications related to the role of 
instructional scaffolding in encouraging shared meaning and learning with WWW resources are 
considered. 



Introduction 

Recent perspectives on learning argue for learning environments that provide opportunities for 
meaningful problem-solving in more realistic contexts than those commonly found in traditional classroom 
settings (Brown, 1992; CTGV, 1992). Many have touted the World Wide Web (WWW) as an 
instructional resource for such complex learning environments due to its unique characteristics to support 
student-directed and collaborative learning (Owston, 1997). The Web allows access to numerous 
information resources on topics such as weather, financial markets, and international news stories. Students 
can use WWW resources to investigate complex problems and to integrate a variety of data sets, 
representations, and perspectives. 

Project-based learning is a pedagogical approach that emphasizes learning through student- 
directed investigation and the creation of shared, public representations of knowledge (Blumenfeld et al., 
1991). Through sharing of project ''artifacts”, learners engage in social interactions aimed toward 
development of understanding (Laffey et al., 1998). Rakes (1996) noted that resource-based learning is a 
pedagogical approach associated with inquiry- and project-based learning in which ". . .students learn from 
his or her own interaction with a wide range of learning resources rather than from class exposition. ..” (p. 
52). Learning with resource-based environments depends heavily upon learners’ mindful application of a 
variety of learning strategies, including information selection, organization, and integration with prior 
knowledge (Moore, 1995; Land & Greene, 2000). Effective use of self-regulation and metacognitive 
strategies are critical to negotiate both emerging conceptual knowledge and the complexity of a vast 
resource base (Windschilt, 1998). 

Yet, Windschilt (1998) argued that two parallel lines of descriptive research are necessary to 
advance our understanding of student learning with the WWW: (a) studies that examine learner cognitive 
engagement in Web-based activities; and (b) studies that examine the types of guidance needed by students 
to effectively learn with the WWW. Accordingly, the purpose of this research was to investigate how 
learners interacted with WWW resources and how they used different forms of instructional guidance, or 
"scaffolding,” to engage an open-ended project task. 
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The Role of Scaffolding in Project-Based Learning Environments 



Project-based learning environments typically involve students in solving complex, real world 
problems that can be approached in different ways and have multiple solutions. These learning 
environments require self-regulation and sophisticated forms of cognitive processing (such as application, 
analysis, and synthesis) that are often problematic for novices who, by definition, do not possess the 
knowledge necessary to solve problems in a new content area. This paradox has been addressed by 
researchers who have shown that instructional scaffolds can successfully support cognitive functioning of 
novice learners, so that they can benefit from complex learning environments (Brown, 1992; Palincsar, 
1986), Scaffolds are instructional procedures designed to support learning so that a student can improve 
beyond his or her current level of understanding with guidance from a peer, teacher, or instructional aid 
(Rosenshine & Meister, 1 992), Often, scaffolds prompt students to induce metacognitive processing 
(Salomon et al,, 1 989), support working memory processes by reducing cognitive load (Lajoie, 1 993), or 
provide access to critical information required to complete a learning task. Regardless of the type of 
scaffold, it must be recognized by the learner as useful and must be flexible enough to engage the learner at 
his or her current level of understanding. 

Social Scaffolding 

Adoption of the metaphor of scaffolding originated with Palinscar and Brown’s (1984) work on 
reciprocal teaching of reading comprehension strategies (Rosenshine & Meister, 1994), One key 
component of reciprocal teaching is that the teacher’s involvement in the instructional process changes as 
students gain competence in strategy use. The teacher initially models the strategies and guides students in 
learning to use them. Teacher guidance decreases as students take turns modeling the strategies for each 
other. The second key component of reciprocal teaching is the role of dialogue (Palincsar, 1 986), Through 
talking with teachers and peers about the meaning of an instructional event, learners are encouraged to 
expand and to internalize new understanding. Learning through dialogue and collaboration has been a 
discemable theme in instructional research since the late 1980’s, In the present study we looked at how 
social scaffolding in the forms of student-teacher and student-student interactions influenced project 
development and informed use of WWW resources. 

Scaffolding Tools Embedded in Instruction 

Not all scaffolds focus on dialogue. Scaffolds can also be woven into learning activities through 
the design of instruction, as when computer technology is used to insert scaffolding devices directly into 
the interface (Lajoie, 1993), For example, Salomon, Globerson, & Guterman (1989) used embedded 
scaffolds with the “Reading Partner” by providing prompts in the form of pull-down menus to help learners 
reflect on their understanding while reading. Similar scaffolds were embedded into the “Writing Partner” 
to guide learner metacognition during the writing process (Salomon, 1993), Hence, technology was used as 
a cognitive and metacognitive tool to support higher-order thinking. 

Embedded scaffolds function neither to supplant learner tasks or cognitive processing nor to 
provide explicit direction. Instead, scaffolds guide learners to reflect on their actions, or they provide 
general suggestions for productive inquiry, Lajoie (1993), for instance, used embedded scaffolds with an 
avionics troubleshooting environment to support memory, metacognition, and hypothesis testing. The 
software tracks the actions of the learner and provides a summary of actions that can be used reflectively to 
consider new possible lines of inquiry. Like Salomon’s ReadingAVriting Partner (Salomon, 1993), the 
embedded scaffolds were used to help learners reach beyond current levels of understanding, 

Scardamalia and her colleagues (1989) have used the term procedural facilitation to describe the 
type of scaffolding investigated by Lajoie (1993) and Salomon (1993), Procedural facilitation typically 
combines support for learner metacognition with the natural capacities of computer systems, such as storing 
and retrieving information, supporting text and diagram formats for structuring knowledge, providing 
menus for choosing action and getting help, and recording sequence of choices and actions (Scardamalia et 
al,, 1989), In general, procedural facilitation encourages metacognition and helps learners focus on the 
important aspects of the learning task. Hence, the notion of procedural facilitation is not limited strictly to 
embedded use with stand-alone computer applications. For example, teachers often use written guidelines 
to help learners plan, reflect, and manage a complex task, such as class projects and assignments. In the 
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present study, we examined use of a written form of procedural scaffolding to help learners manage the 
complexity of their projects and a resource-based environment. 

Purpose and Guiding Questions 

Despite considerable interest in instructional scaffolds, few studies have examined how learners 
use multiple forms of scaffolding during learning. Accordingly, the purpose of this study was to investigate 
how learners used WWW resources and various forms of scaffolding during project-based learning. Using 
qualitative methodology, we examined how varied scaffolds influenced learner development and 
refinement of project ideas. The following questions guided our analysis: 

• How did learners use WWW resources to develop projects? 

• How did a procedural scaffold influence project development? 

• How did student-student interactions influence project development? 

• How did instructor-student interactions influence project development? 

Method 

Eighteen undergraduate pre-service teachers participated in the study (16 female; 2 male) from a 
medium-sized. Southwestern University. The participants chose their own project topics and elected 
whether to work in self-selected groups or by themselves. Eight cases were formed from students choosing 
to work in six different groups of two or three, and two students choosing to work independently. Four 
cases (N=9) were selected for in-depth analysis based on the following criteria: (1) richness of information 
and completeness of videotapes and think-aloud protocols in relation to the research questions; (2) diversity 
of project ideas; and (3) diversity of grouping options (two groups of three; one group of two; and one 
individual). The remaining cases were analyzed informally at the case-level, but in-depth analyses of all 
cases occurred during cross-case interactions. Where participant data are reported, the pseudonyms are 
used. 

The classroom context was a “Learning with Educational Technologies” course that covered a 
variety of broad-based computer technologies. Students were studied during the WWW unit, where they 
were required to develop activities for integrating the WWW into the curriculum. Specifically, students 
were asked to construct a lesson that would involve using the Internet in the classroom. The project task 
entailed three main components: (a) the identification of WWW resources or sites that would form the 
resource base for the activity; (b) the generation of compensatory instructional activities that would guide 
student use of WWW resources; and (c) the completion of written documentation to argue and support 
project ideas. Two instructors (the two authors of this manuscript) participated fully during the teaching of 
the WWW unit. 

The WWW Unit lasted four consecutive days during the summer term, and each class period 
lasted approximately 2.5 hours. On Day 1, students were introduced to using a WWW browser, the project 
requirements, and think-aloud procedure. The remainder of the class time (approximately 1 hour) was spent 
searching the web and generating project ideas. Days two and three were spent entirely on project work 
and web searching. On Day 4, students were assigned to groups to share ideas and feedback with each 
other. All interactions were videotaped, with the camera recording activities on the screen and audio 
recording student conversations. 

The Instructional Scaffolds 

Scaffolding with WWW resources. 

WWW resources served to augment the kinds and amounts of information accessible to students. 
An important goal of the study’s instructional context was to expand preservice teachers’ conceptions of 
how to use computer technology in their future classrooms, considering current perspectives on teaching 
and learning. Our experience has shown that these prospective teachers initially see WWW resources as 
limited to relatively superficial purposes, such as a vast electronic encyclopedia for book report 
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assignments, rather than a multi-faceted resource for open-ended inquiry. As our students searched the 
WWW for ideas on how it could be used in the classroom, we expected them to encounter resources and 
applications that they had not previously considered. Thus, it was assumed that WWW resources could 
provide learners with more efficient access to relevant information (Perkins, 1991) which could 
subsequently “scaffold” their ability to consider new ideas for teaching with technology. 

Procedural scaffolding. 

The procedural scaffold consisted of a series of guiding questions designed to assist students in 
focusing and developing their projects. These questions were provided to students in the form of a 
typewritten handout. The purpose of the questions was to help students frequently evaluate and reflect on 
their project ideas by having them answer what and why questions about their current project plans in order 
to explain their ideas. Some of the questions directly corresponded to required sections of the project 
written documentation. Students were asked to discuss aloud the questions at the beginning and end of 
each session and whenever they thought they were confused. 

Student-student interactions. 

Collaboration with student peers occurred in two contexts. The first form of student-student 
interaction occurred during conversations with group members working together on the formative 
development of their project. The second form of student-student interaction occurred between other 
groups in the class, after projects had been substantially developed. That is, on Day 4 of the study, each of 
the eight cases was assigned to a group consisting of 2-3 cases each (approximately 6 students per group). 
Each case exchanged written documentation of their project with each other and read, discussed, and 
provided suggestions for revision. All students were given a list of guidelines to cue them to important 
areas to provide feedback. 

Instructor-student interactions. 

These interactions consisted of ongoing dialog and conversation between instructors and students 
during project development. The instructors continually circled the room to answer student questions and 
to determine current levels of understanding and project success. Instructors would query students for 
explanations behind decisions and would assist them in re-framing and expanding ideas when necessary. 

Study Design, Data Sources, and Analyses 

The study was based on a multi-case design, as described by Miles and Huberman (1994), where 
multiple cases were studied individually and through cross-case comparison. Two units of analysis were 
applied: (a) analysis of the individual student or group of students who worked on a specific project topic 
(e.g., analysis within the bounded contexts of Case 1, Case 2, Case 3 etc.); or (b) analysis of multiple cases 
interacting with each other to refine a specific case’s draft project (e.g.. Case 1, 2, and 3 working together 
on Day 4 of the study). The design was naturalistic and descriptive, utilizing qualitative research methods 
to analyze development within an authentic classroom setting. The primary data sources consisted of 
videotaped observations of WWW use and transcriptions of associated student verbalizations. Verbal 
protocols were produced from learners “talking aloud” while they used the web and discussed project ideas 
(Ericsson & Simon, 1993). Videotapes fostered microanalysis of how learners used the WWW in 
conjunction with conversations (students and instructors) about their project. Thus, in concert with video 
tapes, the verbal protocols enabled us to examine in detail learner sense-making processes that emerged 
from interaction with instructors, students, technology, and other scaffolds. 



Results 

Data from think-aloud protocols and observations from videotapes were analyzed to determine the 
processes whereby learners engaged WWW resources and the influence of different types of scaffolds on 
learner project development. The findings will be described in terms of each research question. 
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Question 1: How do learners use WWW resources to develop projects? 

One goal of exposing learners to the WWW as a medium for instruction was to expand their 
conceptions of what could be accomplished instructionally in varied or different ways. In order to 
accomplish this, learners needed to first recognize specific resources of the WWW and their potential 
contribution to their project and to teaching and learning practices. Second, learners needed to integrate 
them into their project activity. We found that the extent to which this occurred depended on the degree of 
knowledge learners had about the WWW, the topic they were investigating, and their openness to new 
ideas. There often seemed to be a difference between recognition of WWW resources and integration of 
them into the project. That is, there was a difference between immediate changes to existing ideas and 
lasting changes, in that sometimes students recognized how specific WWW resources could suggest 
changes to their project idea, but the suggested changes were not incorporated. 

Examples from the data. 

Both Case 2 (Tammy and Kara) and Case 3 (Janie) demonstrated instances where they recognized 
and used WWW resources in relation to the project. For instance, when browsing through sites related to 
the stock market, Tammy and Kara (Case 2) found a simulation program with real-time stock market data 
that could be used by their students to learn how to invest. Later on, they found a chat room and 
recognized that it could be used as a resource for communicating with experts. Although Tammy and Kara 
were not very knowledgeable about either the WWW or the stock market, they seemed to approach the 
project with a belief that their searching would lead them to good ideas. Thus, they seemed attuned to 
resources of the WWW that could be used for their project. 

For Janie, however, several of her WWW discoveries were never incorporated into her final 
project, and she seemed to struggle with consolidating her project ideas. That is, she would often recognize 
how WWW resources could be useful for her project, but she did not actually use many of them. This 
seemed to be related to difficulties in delineating the boundaries of her project’s scope and to problems 
with ineffective search strategies. Although she reported high knowledge of the WWW, she demonstrated 
novice search strategies. She browsed through many irrelevant sites, in part, because she failed to clearly 
identify the types of resources she wanted to find in relation to her project goals. Although her inefficient 
search strategies cannot explain all of her consolidation problems, they did contribute to the disconnection 
found among her project ideas and WWW resources. 

Case 1 (Jack, Jane, Jill) was also able to recognize and use WWW resources for their project, 
although in different ways. Case 1 had a very firm idea about how they wanted to use WWW resources to 
support their project idea. This was due largely to the solid background knowledge the group had about 
trip planning, and Jack’s high system knowledge of the WWW and the resources available. The group had 
planned trips before, so they were aware of the types of information that their students would need to plan a 
trip. This helped them identify relevant categories of information that would be needed. Similarly, Jack 
was already aware of numerous WWW resources for travel (e.g., mapping tools; sites to investigate airline 
and rental car rates, etc.), and he was able to draw upon this knowledge to help the group find the kinds of 
resources they wanted. 

Question 2: How does a procedural scaffold influence project development? 

Procedural scaffolding occurred through the use of a typewritten set of guiding questions designed 
to prompt intentional reflection during the project. Students were asked to address the questions at the 
beginning and end of each day and whenever they thought they were confused. We found that the 
procedural scaffold was often insufficient because students sometimes omitted questions and/or answered 
superficially, thereby failing to engage in deeper processing. Rather than use the questions as a tool to aid 
cognition, learners seemed to consider them a restriction to their progress and often ignored them. 

Initially, students commonly viewed the project as the development of an instructional activity 
without consideration of the larger curricular context in which it would fit. The goal of the questions was 
to scaffold their understanding of relevant contextual issues, such as the types of problems that the WWW 
could uniquely address and how their activity would address the instructional problem they identified. It 
was expected that the questions would help students articulate how the planned activities related to the 
instructional goals and to the planned use of technology. It was difficult for most students to couch their 
project ideas within the larger context and integrate the components while also trying to “flesh out” the 
details and assumptions underlying their ideas. Hence, the questions were designed to help learners bring 
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together the various components of their project and to promote alignment of goals, activities, media, and 
outcomes. Yet, when addressing the questions, learners instead tended to repeatedly describe the activity 
itself in answer to most of the questions. Rather than differentiate the varied components of their project, 
students remarked that the questions seemed redundant; instead, their responses seemed redundant and did 
not address the distinctions suggested by the questions. 

Examples from the data. 

On the second day of searching. Case 2 (Tammy and Kara, stock market) still could not address 
the question about what they wanted their students to learn. Kara remarked, "it's hard to answer all these 
questions when you just don't know." Similarly, on the third day of verbalizing responses to the questions, 
Tammy said, "larger instructional context— forget that... These questions are redundant." Case 1 (Jack, 
Jane, & Jill, trip planning) had a similar thought on day 2 when Jill expressed in response to the larger 
instructional context question, "I don't know, I think we should just start working on our project." 

Case 4 (Nell, Desi, Opal, Shakespeare topic), on the other hand, seemed to experience few 
problems with addressing the distinctions among the questions. Alternatively, they struggled with 
inadequate topic and system knowledge, which resulted in use of unproductive search terms and inadequate 
retrieval of WWW resources. The procedural scaffold was not effective in this situation, since it did not 
address their fundamental problems related to WWW searching. Yet, we did see instances where the 
questions stimulated metacognitive awareness of the need for a broader perspective or a more refined take 
on the activity. For example, when responding to the question about the type of cognitive processing that 
would be required of the students, Janie replied "Well, I think I need a little help with that. I'm getting 
frustrated and overwhelmed with all this information right now. I need to sit down and work with [this] 
question ... before I can go on." Often, the situations in which students experienced this metacognitive 
awareness were followed by requests for interaction with one of the instructors. 

Question 3: How do student-student interactions influence project development? 

Student-student scaffolding was examined both within cases (for all but Case 3, since Janie 
worked alone) and across cases. Within-case scaffolding was examined in the verbal protocol data of 
students working with their group on the project. Across-case scaffolding occurred on Day 4 when 
students were asked to join with other groups to give and receive semi-structured feedback on drafts of 
their projects. 

We found that student- student interaction within cases was useful in influencing the development 
of ideas when group members offered suggestions, when they were open to negotiation of ideas and sites to 
access, and when they shared prior experiences. We found that the effectiveness of across-case scaffolding 
seemed to depend on whether learners participated in giving others feedback on what did and did not make 
sense to them. Despite the fact that students were given guidelines to help them focus their feedback, some 
groups nonetheless refrained from giving suggestive feedback to each other. Others, however, provided 
feedback that seemed both in-depth and insightful, thus participating in the teaching function of the 
exercise. 

Examples of within-case scaffolding from the data. 

Case 1 (trip planning) experienced considerable within-case scaffolding. Jack, who emerged as 
the group leader probably due to his knowledge of the WWW, often demonstrated how WWW resources 
would help their project on students planning family vacations. For example, on the first day Jack noted 
“there is a [site] on yahoo that can help you make maps.” On the second day he led the group to a site that 
he had accessed previously and said, “ we can find [homepages for selected] cities here. It should list all the 
cities with websites.” Jane, on the other hand, had little experience with the WWW, but she used her prior 
travel experiences to help the group constrain the project focus. For example, when they were discussing 
the use of budgets she said “1 don’t know what y’all think, but we went to California and it cost $1000 in 
plane fare.” There were several instances when she appropriately brought up her prior experiences to help 
the group develop their project idea. 

For Case 2 (Tammy and Kara, stock market), their different perspectives on teaching and learning 
required them to engage in considerable discussion concerning how the project would develop. Such 
discussions seemed to enhance the development process even though their differences were not always 
directly negotiated or discussed. Kara, for instance, continually made requests to include more direct 
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instruction about the stock market (their project topic). Tammy, on the other hand, valued a more hands- 
on, learner-centered approach where decisions would be made without complete knowledge of the content 
area. The following is an example of the type of discussion that was noted frequently: Interestingly, Kara’s 
verbalizations often encouraged Tammy to explain, clarify, and justify her thinking on the project, which 
seemed to help them both further consolidate the project idea. In other words, being at odds on important 
issues seemed to facilitate a dialectical process in their interaction that helped them develop their ideas. 

There were also instances, however, when differences among group members were ignored, 
resulting in fragmented understanding within the group. For instance, at the end of the second evening. 
Case l(trip planning) still did not have a shared understanding of their topic. When this discrepancy came 
to light, they did not seem concerned. There was little evidence that the group discussed decisions or 
intentions, but rather. Jack directed most of the decisions regarding sites to visit and ideas to be concluded. 
Although Jack’s input may have scaffolded the development of their project idea, his dominance may also 
have hindered the contributions of the other two. Case 4 also showed instances of ignoring each other. In 
particular, it seemed that Nell and Desi often ignored Opal. Unfortunately, Opal was also the most 
knowledgeable about searching and often was ignored as she attempted to direct their otherwise aimless 
searching. 

Examples of across-case scaffolding from the data. 

The verbal protocol data from one of the across-group interactions (Group 2 with Cases 2, 3, & 8) 
provided the best picture of how this type of social scaffolding works, and when it does not work. They 
began by discussing the Case 2 (Tammy & Kara, stock market) project. Tammy asked if the others could 
understand their problem statement and another student, Kelly, from Case 8, responded by trying to 
paraphrase it. She said, “The problem is that students don’t like math and you are trying to make it 
something they can apply to real life, and to help them. I’m trying to put it in a nutshell.” This group 
established a pattern of interacting whereby someone outside of the project being discussed either 
paraphrases part of the project or asks a clarification question, and then a project participant would verify 
or clarify. It seemed that the point was to first create a shared understanding among the group, and then 
help make the project elements clearer and, in some cases, offer suggestions for revisions. Another good 
example came from Kelly after Tammy explained how the students would work with EduStock. She said, 
“that answers this [part of the] problem. State that [more explicitly] . . . You might need to include more 
explicit [information] about why the Web is the more effective than using the Wall St Journal.” This 
instance shows how one student helped to scaffold another student’s project development by providing 
suggestions and pointing to incomplete areas of the draft. 

The across-case group involving Cases 4 and 6 was different in that they did not attempt to 
establish a shared meaning before they began giving feedback. Initially most of the feedback was fairly 
superficial. For example, one student, Julie, said to Case 4 (Shakespeare topic) “The procedures are in 
paragraph form, and though I still [understood] them, the format [is problematic]. I think you need to break 
it down [by steps].” Later, Desi’s (Case 4, Shakespeare) first question to Case 6 was “What about printouts 
of web pages?” Yet, both Desi and Opal saw that an element in the Case 6 project, a teacher workshop on 
the WWW prior to starting the project with 5* grade students, was something that could be adapted within 
their project. Also, the low- level feedback gave way to more substantive questions that required 
clarification. The data showed that both cases experienced ftirther consolidation during the interaction. 
Although it took them longer to establish shared understandings of their respective projects. 

The third across-case group, involving Cases 1, 5, and 7, did not engage in conversation across 
cases. Both instructors inquired several times as to what they were doing and prompted them to discuss 
each other’s work. Each time, they said they were reading the drafts, and that they would get to the 
discussion when they were done. It is possible that the group was not comfortable providing critical or 
specific feedback to each other, as evidenced by their shared approach to provide written feedback on the 
paper drafts, without directly interacting with each other. Interestingly, when they finally, yet briefly, 
conversed across groups, they made only laudatory comments while exchanging papers. 

Question 4; How do instructor-student interactions influence project development? 

Instructor-student interactions occurred as the two instructors moved through the classroom 
interacting with the students. The dynamic nature of personal interactions with instructors made it difficult 
for students to ignore the confusion and inconsistencies in their conceptions. In addition to responding to 
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questions, the instructors would ask each individual or group to describe their current conceptualizations 
several times during a session. The purpose of these interactions was to encourage reflection on how the 
developing project created meaningfiil contexts that could be uniquely addressed by the Web. This form of 
scaffolding seemed to be very effective for all but Case 4. 

Examples from the data. 

One of the instructors asked Jack, Jane, and Jill (Case 1) to describe the problem their project was 
addressing. Jill responded by describing the activities in the project. The instructor said "that's basically 
what it is about, but why that? How does it address learning? Why would you use that?" Jack replied, "To 
get them involved personally, like problem-based learning, it makes it more real for them." The instructor 
then responded, "Then why might you want them to, take math for instance, make this more real?" Jill 
said, "Because kids say, when will I ever use this again?" In this instance, student-instructor interaction 
served to assist students in re-framing and strengthening the reasoning underlying their activity, rather than 
simply acknowledging the activity itself 

Case 4 was the only case that did not seem to benefit from interactions with the two instructors. In 
hindsight, we found that we were not successful during the course of instruction in diagnosing their needs. 
Analysis of the verbal protocol data indicated how frustrated this group was with searching the WWW. 
Consequently, this frustration often manifested as tense and curt interactions with each other and with the 
instructors. Although it was clear the group was frustrated, it was less clear what was required to help them 
solve the problem. Despite instructor queries regarding search techniques and terms, the instructors and 
students were unable to reach a shared understanding of the problem. Consequently, instructor suggestions 
to help the group focus their topic and WWW searching were largely ineffective. 

Another example of the problems with the instructor-student scaffolding with Case 4 occurred as 
Instructor 2 tried to encourage the participants to think about what their future students would be thinking 
and learning. 

12: Why the chat room, for instance. Why are they using that? 

Desi: To talk to their peers. 

Opal: [annoyed] Because it is fun. 

Desi: We found a chat line that had different rooms, and like you go into authors, and it has all 

of the different authors listed, and it tells you all of the people in that room. So it might 
have... two other people there who might want to talk about it. 

12: So what do you think they are talking about? I mean, are they talking about “what did the 

play mean to me”, or “how do I make sense out of it?” 

Opal: Huh? 

Nell: As far as their peers, they can find out what people there say about Shakespeare, or if they 

have a question, they can go ask. 

Desi: I think that chat rooms are a cool idea, but I’m not completely sold on it. 

12: No, I’m not suggesting that [you abandon the idea]. I’m just trying to have you think about 

what they will be thinking about. What does this mean for the class? Or what does it 
mean... 

Nell: So their own interpretations? 

One of the interesting problems with scaffolding, that the above quote captures, is that in order for 
scaffolding to work the instructor needs to meet the learners at their current level. It was not until we 
analyzed the verbal protocol data that we had any real insight into where Case 4 was stymied. We knew 
they were having trouble, their level of frustration was quite apparent, but we were unable to figure out 
what they needed to move beyond their frustration. Fortunately, they were able complete the project and 
do adequate job, relying upon the WWW resources they did frnd. It was not, however, a positive or very 
productive learning experience for them. 

General Themes and Discussion 

These findings suggest that WWW resources and the other instructional scaffolds were of some 
benefit to three of the four cases. Additionally, social scaffolding, especially when provided by the 
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instructors, was particularly beneficial for moving students’ thinking beyond a concrete focus on activities 
to a higher level conceptualization of the goals of their projects. We believe there are two general themes 
that emerged from our data: (a) the critical need for scaffolds to be accessible to learners; and (b) the need 
for dynamic, social scaffolding. 

Accessibility of Scaffolds 

The utility of a procedural scaffold, such as our guiding questions, is driven by its relevance and 
the degree to which it is accessible conceptually to the learner (Hannafin, Land, & Oliver, 1999). We 
found that, while the procedural scaffold assisted learners in making their ideas overt, they nonetheless 
were of limited usefulness in supporting evaluation of the reasoning behind an idea. Instead, learners 
tended to overlook inconsistencies in their thinking and often did not use the guidance as a way to evaluate 
ideas. It is possible that these resources were not seen as relevant to the task, but more likely, the kinds of 
processing that were being prompted were difficult to access for the novice learner without the guidance of 
the instructors. The learners were too enmeshed in sorting out the details of the project activities to even 
recognize what the questions were asking of them. They were unable to access the underlying principles 
without explanation, encouragement, and modeling from the instructors. 

We expected that students would encounter resources on the WWW that would expand their 
thinking on an instructional approach that was unique to the Web. This expectation was met for Case 1, 
whose trip planning topic was well supported by numerous WWW resources that were easily accessible, 
and for Case 2, who found an educational website (EduStock that included accessible resources for their 
stock market topic). The other students were left to discover and infer how the WWW resources they 
encountered might aid the development of their project. Although this level of discovery is a typical 
expectation in open learning environments, our data suggest that without other, more overt forms of 
scaffolding, the novice learner will not discover the utility and, therefore, will not be able to take advantage 
of what the environment affords. Although Cases 3 and 4 had flashes of recognition of how a resource 
might expand their instructional approach, they were unable to incorporate those flashes into concrete ideas 
that were included in their final projects. 

The importance of dynamic, social scaffolding 

A significant inference from the findings of this study is the importance of dynamic, social 
scaffolding for learning. By dynamic, we mean real time, back and forth discussion that does not allow for 
ignoring confusion, but rather confronts confusion and inconsistencies in thinking. The learning 
environment that we studied involved the use of a static, procedural scaffold, the many resources available 
on the WWW, and social (instructor and peer feedback) interaction. However, our learners were often 
unsuccessful in using the procedural scaffold or WWW resources to expand their current thinking, probably 
because these resources did not prevent them from proceeding in face of confusion and inconsistencies. It 
was during dialogues, especially with instructors or peers outside of their group, when learners were able to 
think more abstractly and engage in more reflection and reasoning about their project ideas. 

We noted that for dynamic scaffolding to be successful, it was critical that participants overtly 
developed shared meaning. In fact, the process of establishing a shared understanding seem to instigate 
higher levels of cognitive engagement, as students clarified and modified their thinking in response to 
questions and comments from others. It appears that in order to ensure that the more subtle scaffolds, such 
as our procedural scaffold and WWW resources are effective, dynamic interaction, specifically with an 
instructor or a “more capable peer” (Vygotsky, 1978) is needed. In this study the instructor was able, in 
most cases, to prompt reflection through questions similar to the static ones and to encourage the evaluation 
of Web resources as potential resources. But, unlike the static, procedural scaffold, or the WWW 
resources, the instructor was better able to diagnose the limitations and strengths in student thinking. Once 
limitations were identified, the instructor was able to recast or reframe questions, provide examples, or 
offer suggestions. Instructor (or d>mamic) interaction was significant in helping students expand, formalize, 
and refine reasoning behind ideas. These findings are consistent with Palinscar’s arguments (1986) 
concerning the importance of dialogue between students and their teachers during the process of learning. 
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Abstract. 

The study reports the impact of a technology course on in-service teachers. Teachers' comfort 
level, belief confidence, and attitude toward the use of technology were investigated. The impact 
of the course with different degree of technology integration was also researched. The results 
indicated that the course increased students' comfort level, confidence, and attitude toward the use 
of technology. The findings also indicated that no significant difference was found on their 
comfort level, confidence, and attitude between the two groups who had different degree of 
emphasis on technology integration. 



Introduction 

In September of 1997, the National Council for Accreditation of Teacher Education (NCATE) 
released a report addressing the importance of teachers integrating technology into instruction. The new 
technology standards clearly state that teachers must be competent of using technology in their teaching. 

At present, training teachers the use of technology has become a strong nation-wide movement. 

Many universities are currently offering a technology course, often called ’’the technology course”. 
Content of the course and attitude of teachers toward the use of technology have been studied. Research 
revealed that the course content is mostly skill-based and that technology integration should be emphasized 
in the course. Professionals also promoted technology infusion. In addition, research noted that teachers 
need technology training and that many of the teachers are anxious of the use of technology in the 
classroom. 

Does the technology course reduce teachers’ anxiety? Do teachers feel more comfortable and 
confident in the use of technology after they take the technology course? Does the course change their 
attitude toward technology? Do they believe that technology is useful for improving their teaching? Is the 
course content useful to teachers and applicable to their current and future teaching? Does the technology 
course with different emphasis on technology integration or technology infusion have different impact on 
teachers? 

The present study is intended to examine the impact of the technology course on in-service 
teachers. Teachers' comfort level, belief, confidence, and attitude toward the use of technology are 
investigated. The impact of the course v^th different emphasis on technology integration or technology 
infusion is also researched. 

Research Procedures and Methodology 

Four classes, a total of 68 teachers who were taking the technology course in 1999 at a public 
university, were selected for the study. At this university, students who want to clear their teaching 
credential are required to take two educational technology courses, level I and level II. The level I course 
covers computer productivity tools such as word processing, spreadsheet, and database. The level II course 
involves multimedia, Internet, and technology integration. The course of this research study was a 
technology level II course. The majority of the students were full-time in-service teachers. The technology 
course lasted for ten weeks 

Two professors taught this course to two classes each. Based on the professor’s emphasis on the 
integration, two classes were categorized into "Integration A” classes while the other two classes fell into 
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the *' Integration B" category. The content, emphasis, and course projects will be introduced in the section 
“The two categories of the courses”. 

At the end of the quarter, the researcher went to the four classes and asked the students to fill in a 
survey. The survey contained ten Liker-scale questions related to the participants’ perceptions of their 
comfort level, confidence, belief, and attitude toward the use of technology in classroom before and after 
the course. The internal reliability of the survey was checked by SAS computer program. The Cronbach 
Coefficient alpha value of the survey was 0.85. The value was high, and the items of the survey were 
reliable. 

The means of the ten question items were used to determine how strong the students agreed with 
statements that the course was useful and applicable and that technology can improve teaching. Three 
question items were used to examine via t-test possible differences between the two groups, the 
’’Integration A” group and the "Integration B’’ group. 



The Two Categories of the Courses 

The professor of the "Integration A” classes believed that students in the course needed to learn 
computer applications that were commonly used in schools and should practically explore how these 
applications could benefit and be integrated into their teaching. She taught students how to use a 
multimedia program (HyperStudio) and a webpage editor (Netscape Communicator). Students used these 
two applications to develop two course projects — a HyperStudio project and a webpage project. Examples 
of the HyperStudio project (see Figure 1) are: using the software to (1 ) create an instruction on a subject 
area like mathematics, (2) develop practice items, or (3) produce a test. K1 2 learners, especially remedial 
students, could review what they had learned by going through the instructional HyperStudio Stack. K12 
students could also conduct self-examination using the practice items. 




Figure I . Example of as tudent's HyperStudio card. 

In-service teachers could use the test to evaluate their students. For the webpage project (see 
Figure 2), the professor asked her students to write a paragraph of their educational philosophy, insert 
pictures (optional), link to the websites they selected, annotate the websites, and create Netsearch 
questions. The selected websites should relate to their subject areas (e.g. language art, mathematics, 
science, or social science). The Netsearch questions, like Scavenger Hunt, should encourage their students 
to search for the answers on the Internet. To motivate students’ learning, the students conducted peer 
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evaluation on all their webpages on the Internet or local computers (see Figure 3). The other two course 
projects were a research paper researching on availability of hardware, software, and teachers’ use of 
technology in their schools and a lesson plan integrating technology into instruction. 



^ ^ File ECHt View Go Pooianarics Communicator Help 



12:18PM Netscape Communlcotoi** 



I Netscape: E 



^ ^ 

awk f'vrrfirii Rftottf 



A 



S«jnsti N»tto«p* 



m. 



PrW 



S«our<ty '-Shcp 



Laoition: |hHp ://«o».eturi>>du/*tok/rrEC346_vt<1»r99/lna»x.html 






Vhat't RaUUd 



ETEC : : 
- InU-pdu^Qg 



S^labus -' '• 



Home " . 



ws 



Hi 






Curtiss and Kelly's Homepage 
Welcome to Our World! 

'4 A .k '4 4 4 ;A 4 4 4 4 4 4^44444 4 4 44444 4 












/| - Kdly toves her poauon trarfcng ir,bih at C«) on High School' 

■ iaSmfienordmo. She^it C«q on's newesi addition to their ' ' J 

.T. Intemahonal Baccalaureate Faculty* She is teaching * IS. > r ' 

* ' ; t Geometry* and ''LB. Studies' 10 Ceos's Best and. . . ,.'v 

y^Biightest studeotsi She ogoys golf, teomvhorsriack 
' ;_r < 'i anil moonlit walks throu^i the oiai^ groves , r' . , , , . •' ■ 

-■'a: Ci^ss also loves his poation tmchiag Mua'c at RcdUmis; - ' ' * 
'High School. He Oirects the Terrier Marct^ Band. 'A^nd- . 
Cxsemble; Synqihomc Band, and Jhzz EnsanU& He ^sa^ \ r^: 

* v-Yte***>**P*®^® Muac-Theoryinadditioato 

. . . . ovcrseemg the acbvities o f the RHS Percusaon nnsembi e and ^ ^ 

RHS Winterguanl fi oth Kdly and Curtiss are studying for; ' 

* /'tbdr hbter^De^ees at Cal State San Bem^dino and are. - 

y ' • - Imq^ raiang a.small doi^ F enny. . :>■. ^ 

* Follow the Jmte below to view some of cur fhvorite alei:-? ” " 

Browse our other pages to check cut lomerial which diculd s 
prove htdpAd to muac and uaih educators and sbidslsalikd'. 

. : ■ .Once you've looked around,- give us a hollar on the snail and ;* 

* Vlctusknowwhalyouthinkor jua say howly! Keep the ■,•'.* 






}S 

■Si 

’M 










Figure 2. Example of a student's webpage. 




Back -Ff-rvird R^loid Home Soarnh Netsoape. V Print ’ Seck^ Stop 



Location : jhttp ://soo. crush ■tdu/alah/ETEC346-i^l98/rrwtth»ttud»nts.htm 



Vhat's RaUiod 



5ETEG 

1nu^t^on > 



>4,3^ ir' 






T3.r-¥vSB: 







vCUcXonJ^ 'hMQ8 of ,&^^eirt:to\vie yebpagwT^ 




^ Lisa' Di^ Ahuero= . ; Wrf ia Ter^ A1 vai^ •' Kiifcaotos Ari ' 



‘ O^'; piMie Ahuero= . ; Maria Tei^ A1 var^ " Milaj^os An Bedl^ter 
;X' :, ' . ‘ Sean AmiefV Bn^ > WilUajtn C: Bro^ .; .; Kaihl,Mn-ManG 

Eugene Nile Rawer, r v .Kenneth Phil Ingr^ ' • • Douglas Edvard^ Jacobs 
: r ■ Ywe^ .C^ol:Xhra An^laB^ph- y/TymXCnstinM 

^ :Ery 'beah Diz^b Petef sen V^ ■ • Kathy^t^^ 

7 .'\ "• Eldon' B. Rvnders-^ ; ./r^'"^SharoX:I^aS^^ ^v- ' • ■? f- M^k A 



H»-r~ 



vig- 



193 

BEST COPY AVAILABLE 



199 



Figure 3. WebDa 2 e projects for veer evaluation. 



The professor of the "Integration B” classes believed that students of the course needed to learn 
infusing technology into subject areas and hoped that after the infusion one could not distinguish 
technology components from subject components. The course projects involved writing a lesson plan, 
evaluating four pieces of educational software, using KidPix to create a slide show illustrating the use of 
subject areas, as well as writing a paper reviewing articles on technology integration. For the lesson plan 
project, students were encouraged to integrate technology like spreadsheet, videodisc, database, or CD- 
ROM into their content areas and grade levels. For the slide show project, students were required to 
include pictures, text, and sound in the project. For the paper, students should list three implications of 
technology integration. 

Both professors required students to write lesson plans integrating technology into instruction. 
Both professors also required students to conduct research. The research of students in the professor A’s 
classes was to investigate the availability of hardware and software of their schools and teachers’ use of 
technology at the schools. The students needed to collect, analyze, and report data. The students in the 
professor B’s classes conducted library research — reviewing literature related to technology integration. 

Although both professors had their students integrate technology into instruction, the degree of the 
integration covered in the course was different. The students of professor A’s classes spent more time than 
the students of professor B’s classes on learning computer applications, like HyperStudio and webpage 
development. The students of professor B’s classes spent more time on technology integration by writing 
the research paper and evaluating software. 




194 



200 



Results and Discussion 



Items 


Means 
of A 


Means 

ofB 


Means of 
A&B 


What I learned from this class is useful to me. 


4.2 


4.2 


4.2 


I can apply what I learned from this class in my 
current classroom. 


3.7 


3.7 


3.7 


I can apply what I learn from this class in my 
future classroom. 


4.1 


4.2 


4.2 


Before I took this class, I felt comfortable of using 
technology in my classroom. 


3.4 


2.9 


3.2 


This class increased my comfort level of using 
technology in my classroom. 


3.8 


3.9 


3.9 


Before I took this class, I felt confident of using 
technology in my classroom. 


3.4 


2.9 


3.2 


This class increased my confidence of using 
technology in my classroom. 


3.8 


3.7 


3.8 


Before I took this class, I had positive attitude 
toward using technology in education. 


4.1 


3.4 


3.8 


This class increased my positive attitude toward using 
technology in education. 


4.1 


4 


4.1 


I believe that technology is useful for improving 
my teaching. 


4.4 


4.4 


4.4 



Table 1. Means on the survey. Means of A = Means of the 34 participants of the professor A*s classes. 
Means ofB = Means of the 34 participants of the professor B*s classes. Means of A & B = Means of all the 
68 participants. 

The results (see Table 1) showed that the participants thought that the course was useful (M=4.2) 
and that they could apply what they learned from the course in their current instruction (M=3.7) as well as 
future classrooms (M=4.2). The results indicated that the course content was appropriate, and that the 
teacher training was on the right track. Some teachers expressed that they could not apply what they 
learned in their current teaching because computers at their schools were inadequate or not available in 
their classrooms. However, they believed that they could apply what they learned from the class in their 
future classrooms. 

The results showed that the students were moderately comfortable (M=3.2) and confident (M=3.2) 
of using technology but had a comparatively positive attitude (M=3.8) before they took the course. This 
indicated that the teachers were willing to use technology although they were not proficient in using it. The 
course increased their comfort level (from 3.2 to 3.9), confidence (from 3.2 to 3.8), and positive attitude 
(from 3.8 to 4. 1). As we know, technology is constantly changing, and the content of the technology 
course is being continually adjusted. Therefore, in the technology course helping teachers feel comfortable, 
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confident, and positive is as important as teaching them specific technology content. If our teachers are 
comfortable and confident in using technology, they will be more likely to succeed in overcoming obstacles 
and using technology in their classrooms. 

Professor A’s class responses on comfort level, confidence, and attitude after the course were 
compared with those of professor B’s classes. The obtained t-values were 0.12, 0.08, and 0.24. The values 
did not exceed the critical value 1 .98 for p=0.05. The results indicated that there was no significant 
difference between the two groups of different emphasis in the course. 

Educators have been debating on what the course should emphasize: technology skills, technology 
integration, or technology infusion. Some educators think that the course should emphasize skills because 
one cannot integrate technology without having the skills. Some educators think that the course should 
focus on technology infusion because teachers must be able to use technology within their content areas 
and grade levels. Probably all instructors of the technology course have currently included technology 
skills and integration in their courses. Degrees of technology integration and ways of integrating 
technology into instruction might vary from one instructor to another. The findings of the study indicate 
that the difference is not significant between the two groups on students' comfort level, confidence, and 
attitude toward using technology. 

The participants also believed that technology was useful for improving instruction (M=4.4). 

Since computers entered our classrooms, educators have been engaging in a constant debate whether 
technology or computers were useful in instruction. Educational technologists often advocated benefits of 
using technology in education. Some professionals voiced their doubts because research indicated that 
technology did not have impact on students' learning. They argued that instructors did not need to use 
technology because it did not enhance students' learning. The results of the study revealed that field 
teachers and educational technologists were heading toward the same direction. Both of them considered 
technology to be useful for improving teaching. 

The study was limited because of the lack of pre-survey and limited number of survey questions. 
The research could be improved by having a pre-survey or more survey questions concerning participants’ 
comfort level, confidence, and attitudes. More research is needed in this area to generate a conclusion. 



Conclusion 

Studies on the technology course at Colleges of Education has been conducted by many 
researchers with the attempt to find the best way to train our teachers to be technology literate in order to 
enhance quality of education. The results of this study showed that the technology course increased 
students' comfort level, confidence, and attitude toward the use of technology. The results also indicated 
that no significant difference was found on students' comfort level, confidence, and attitude between the 
two groups who received different training in integrating technology into instruction. More studies need to 
be conducted to evaluate the impact of the course on teachers’ training. Well-trained teachers will help the 
coming generation to be successful learners in the Information Age. 
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Effects of Learning via Web Design Approach on the 

Students’ Perception 



Byungro Lim 
ByungMin Yu 

Indiana University at Bloomington 



Hypermedia and the Internet have lots of potential in education and training setting. The 
technologies help to create reflective learner who can apply knowledge and skills to novel and real-world 
situations (Perkins, 1992). Using hypermedia and the Internet, learners become engaged in meaningful 
learning instead of rote memory of the discrete concepts. On the other hand, the constructivist perspective, 
knowledge is constructed by learners through interactions with the environment, guides the way the 
technology is used (Jonassen, 1991 ; Duffy & Jonassen, 1992). Learners are expected to construct their own 
meaning, and that meaning is inextricably linked to their cultural history and the context in which they are 
learning. In the constructivist classroom, teaching should be directed toward open-ended inquiry to 
encourage creative reflection on one’s own experience (Scheurman, 1998). 

In this regard, introducing learners to hypermedia is reported to be highly successful in supporting 
learning (e.g., Asyerman, 1996; Lehrer, 1993). It requires students’ active participation and engages them 
in authentic problem investigations. One of the advantages of hypermedia is that its intrinsic design allows 
for linking by association inherent in the information (Smith, 1993). Femlund & Cooper-Shoup (1991) 
asserted that, when students create hypermedia products (in a student-as-hypermedia designer approach), 
they should search for information, arrange and organize the information appropriately; students should 
also explore relationships of the information in new ways. These authors also assume that using 
hypermedia is likely to give students a unique opportunity to develop information retrieval and information 
processing skills. 

Research shows that in the student-as-hypermedia designer approach, students are highly satisfied 
with the course and develop skills and knowledge effectively (Micaise & Crane, 1999). Research also 
indicates that a student-as-hypermedia designer approach encourages students to assume a greater 
responsibility for learning. Prior researchers have documented that students become heavily invested in 
their projects, and on occasion, students spend entire weekends at school working on projects (Lehrer, 
1993; Liu & Pedersen, 1998). 

Even though many authors indicate that the learning by design approach in using hypermedia is 
effective, there is few research regarding students’ perception and effectiveness of learning by web design 
approach. Just like students learn content and design skills through hypermedia design, learning by web 
design approach uses the web design as primary vehicle of learning. Students will search the relevant 
information, and design and develop the web pages. In this process, they should acknowledge the merits 
and weaknesses of the web and find out the best way to communicate their audiences. 

The purpose of this study is to explore students’ p'erce'ptidh about learning by web design 
approach and identify some useful recommendations to which an instructor can refer who is willing to use 
the approach. The research questions include: 

• How do learners feel when they are doing the web design project? 

• Can web design activities, e.g., determining layout or images motivate learners? 

• Do learners feel that they achieve learning objectives after finishing the project? 

• What are merits and weaknesses of learning by web design? 
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Methods 



Participants 

The subjects of this study were graduate students who developed web pages as a main assignment 
for an educational psychology class in a mid-western university. Starting from fall semester 1997, the 
instructor assigned students to develop web pages that introduced educational psychology or provided 
course information. Students could select it instead of writing final paper. At the first semester of 1997, six 
students selected the web page building as their main assignment. Their specific topic was related to history 
and issues regarding intelligence theories. They needed to search and organize information, and design and 
develop a web site dealing with intelligent theories. Since the project was not finished within the semester, 
several students were engaged in this project during the second semester of 1997. The final version can be 
seen at http://www.indiana.edu/~intell . From the first semester of 1998, some students started developing 
the course home page as their main assignment. It included the topics regarding educational psychology as 
well as course information. The final version can be seen at http://www.indiana.edu/~edpsvch . 

In total, twelve students selected this activity as their main assignment during the four semesters 
(1997-1998). Subjects’ academic areas were mainly educational technology and educational psychology. 
Half of them were competent with web authoring skills and a few did not know how to make web pages. 
There were 5 female and 6 males. In most cases, students were grouped for the assignment and the roles in 
a group were divided by their competency and interests, e.g., web developer and information provider. 

Instrumentation 

Initially, data were collected by questionnaire via e-mail. Since each subject had one’s own unique 
email account and regularly used e-mail as a communication tool, using e-mail for the questionnaire was 
regarded as an effective way to collect data on time. The purpose of the questionnaire was to collect 
quantitative data regarding the subjects’ feelings and attitudes toward learning via web design approach. 
The eight questions were asked including “Are you competent with web page authoring?”, “Is the 
assignment is appropriate and effective in terms of learning the content?”, “Do you feel comfortable when 
you were doing the web production assignment?”, and so forth. 

Each question had a Likert scale ranged from ‘Strongly Agree’ (5) to ‘Strongly Disagree’ (1). 
Under each question, there was a space for detailed comments from the subjects. In addition to the 
questions, there was a section of ‘general comment.’ Eleven of twelve subjects responded to the e-mail 
questionnaire. For the subjects who made unclear or unusual answers, a second round questionnaire was 
conducted in order to gather more detailed answers. It was conducted for the limited number of subjects (4 
persons) and specific items (usually 2-4 questions). For example, one student replied without any 
comment that there were few detracting factors during web production assignment, which was unusual. So 
the research asked him comment on his answer in second round questionnaire. 

Data Collection 

Based on the questionnaire data, the researcher selected interviewees for further information. The 
purpose of the interview is to identify the subjects’ feeling and perception in detail which could not be 
revealed by questionnaire method. To get the more representative samples, the researcher selected the 
interviewees across the web competency level (low: 3, high: 3)~and motivation level (low: 3, “high: 3). By 
selecting diverse interviewees, the researcher was able to yield more representative data, which were more 
credible. 

A semi-structured Interview method was adopted for the interview. The interviews were audio- 
taped with the consent of the interviewees and transcribed later. Some interview items are “How did you 
get the content for the web page? How did you organize the content for the web design?”, “During web 
page production, what difficulties did you face?”, “How did you feel when you were doing the web design 
project?”, and so forth. 

These questions show what the subjects felt when they were developing web pages and how they 
valued the activities. When the researcher asked specific items related to web pages, the subjects often 
showed their web pages on computer screen. The researcher then reviewed the web pages and asked the 
subjects where they did get the information, what kinds of difficulties they were faced, what were the main 
focus of them, and so on. These data were also audio-taped as a part of interview and transcribed later. 
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Data Analysis 

For the analysis of quantitative data from the questionnaires, correlation analysis was used. After 
identifying the significant correlations (at .05 level) between variables, the researcher conducted 
independent t-test for the variables. As independent variables, web skills, content organization, and 
detracting factors were used since they were considered as main factors to affect student^s perception. 
Learners’ perception was supposed to be measured in this study. As dependent variables, perception of 
effectiveness of the project, their feeling about comfort and learning, and motivation degree were used. 

Open comments of questionnaire and interview data were organized and categorized around the 
topics; effects of group activity on students’ perception; perception about effectiveness of the design 
approach; motivation issue; aesthetics concern. Multiple data sources (questionnaires, interview, and web 
pages review) were used to draw inferences. The interviewee profiles were used to get representative 
samples. The data analysis was done by descriptive statistics, correlation analysis, and qualitative method. 
The findings were shared with an expert researcher and some interviewees to verify and clarify the 
findings. As a result, the reliability of this study was established. 



Results 

Quantitative data showed the overall satisfaction was high with the web design activity. In general, 
student considered this project as effective (the mean is 4.0 out of 5.0), they felt comfortable when they 
were doing the project (the mean is 4.36 out of 5.0), and they felt they learned a lot (the mean is 4.09 out of 
5.0). 

Table I . Descriptive statistics of each question 



Questions 


Mean 


S.D 


N 


1 . Are you competent with web page authoring? 


3.45 


1.27 


11 


2. The assignment is appropriate and effective 


4.00 


1.26 


11 


3. Do you feel comfortable when doing web design activities 


4.36 


.81 


11 


4. The content was well-structured, so no need to put much efforts 


2.80 


1.40 


10 


5. The publicity of the web affects the web design activities 


3.09 


1.51 


11 


6. Learning content from web design activities was great 


4.09 


1.30 


11 


7. Many detracting factors exist during the web design activities 


2.60 


1.43 


10 


8. Uncomfortable with poor appearance of web pages 


4.10 


1.10 


10 



The correlations analysis reveals the positive correlation between the questions: ‘did the publicity 
of web, i.e., everyone can see the web pages, affect your job?’ and ‘do you feel comfortable when doing 
web design?’ ( r(l 1) = .624,/?<.05). Subjects who were motivated by the publicity of web tended to feel 
more comfortable than the others when doing web design activities. That is, subjects who valued the 
publicity of the web tended to have more positive attitude toward the web design approach than the others. 

Open-ended comments in the questionnaire and interview data were used for the qualitative data 
analysis. The findings are organized around the themes: students’ expectation of the web design 
assignment; the effects of group activity on students’ perception; the effects of content structure on 
student’s perception; perception about effectiveness of the design approach; motivation issue; aesthetics 
concern; other issues. 

The effects of group activity on students’ perception 

The web design assignments were mostly done by group project. Many students had positive 
attitude toward group activities in doing web design. For example, a student who had low level web skills 
felt intimidated at first since she “ had no idea how to go about the project.” She could do well because 
other “graduate student helping me was excellent & I quickly realized that it’s something I could do, 
without too much hassle.” On the other hand, there were some issues regarding group activities. A student 
replied that we was very uncomfortable because she “worked with a team of classmates, one of who was a 
“techy”.” Also, students tended to focus only on their role in the group. Admitting that her group was 




199 



205 



greatly helpful, a student replied that she “was only responsible for collection the intelligence background. 
About the web site work, I don’t know! !!” 

The effects of content structure on students’ perception and learning 

The contents that the students dealt with when doing web design are likely to affect students’ 
satisfaction. If the content was well-organized and easily found from the libraries or web pages, students 
did not think that they had learned a lot from the project. Students reported that they learned a lot when 
dealing with the ill-structured content. A student who put lots of efforts to search for the information, 
replied; “The information existed in several sources, I put my own structure to the info though - 1 didn’t 
just copy the info. I synthesized and organize ... I learned quite a bit about the specific topic I did a hot 
topic on but that was a little narrow for me- 1 would have preferred to focus on a topic that was a little more 
broad.” 

Perception about effectiveness of the design approach 

Many students indicated that they enjoyed the web design assignment. As an assignment, a student 
indicated, it is “even better than a more traditional paper” because it is more “hands-on.” It also facilitates 
students’ thinking. For example, a student was satisfied with the assignment because “it forced me to 
clarify the subtle issues that I would otherwise just let go.” Another student valued the project since “it 
pushed me to think hard about the “boundaries” of the theories when I tried to organize them.” 

Students’ feeling and satisfaction toward the web design activities seems to be determined by 
personal needs and web design skills. One who focused on design itself did express dissatisfaction about 
the project. “Since the project was largely design oriented and not content oriented, I did not feel it was 
useful from the standpoint of the course.” A student who mainly focused on developing web pages said, “I 
think it[leaming content as well as design skills] will vary depending on the goal. In my case, my goal was 
directly related to the web and effective in getting the design skills. But my project didn’t further my 
understanding of the topics.” On the other hand, some students liked to learn web skills themselves. A 
student who lacked in web design skills was satisfied because “the assignment definitely helped me gain 
understanding/confidence in creating web pages.” 

The publicity of the web and motivation 

A feature of the web authoring activity is the publicity of web. Everyone in the world can 
see the web pages that students make. It is reported that the publicity of the web affects students’ 
motivation (Asyerman, 1996). In this study, many students from the interview indicated that the idea that 
the web site would be seen and used by other people motivated them. A student put much effort on the web 
project “because it would be seen and used by others. I was even more committed than usual to have 
accurate information that was clearly written and easy to understand. Also I tried to note/state connections 
to other ideas/people on the site instead inferring the connection.” 

Aesthetic concern 

Many students replied that aesthetic aspect of web design was important “since the primary 
method of delivery in tech www is visual.” A student who was a teacher replied that she would feel “very 
uncomfortable about producing a web site that was’riot professional looking because I understand how 
important visual information is to learning.” Some students who mainly focused on web development 
replied that it was important to create a usable and appealing visual interface for the site. 

Discussion 

Despite some conflicting remarks, nearly all students were satisfied with designing and creating 
their own web page. This study shows that using the web as a tool for knowledge construction is valuable 
and effective from the students’ point of view. As Harel & Papert (1990) indicate, to place the students in 
the role of instructional software designers is an effective way to promote their learning. The findings show 
that most students are highly comfortable with the web design project and felt they learned a lot during the 
project. From the findings, some recommendations about web design project are followed. 
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First, the reason to use the web design approach is that learning by doing approach in general 
helps students learn the content in authentic ways. To be most effective, the web design approach should be 
supported by sound instructional strategies. Motivation strategies, teaching and learning mode, feedback 
strategies, and evaluation methods should be carefully prepared. 

Second, effective group strategies such as active group discussion, evaluation criteria, effective 
group management should be carefully prepared. One of the limitations of the learning by design approach 
is that it mainly focuses on the narrow knowledge (Nicaise, 1999). In the web design approach, if there is 
no mechanism to facilitate group discussion, students tend to focus on narrow knowledge rather than wide. 
For example, a student indicated that “I felt that I learned quite a bit about the specific topic I did a hot 
topic on but that was a little narrow for me — I would have preferred to focus on a topic that was a little 
more broad.” 

Carefully prepared group activities and flexible role division among group members make a 
difference. Fixed role division does not help the students learn both the content and design skills. For 
example, a student just focused on the seeking information. She said, ‘T was only responsible for collection 
the intelligence background. About the web site work, I don’t know! I” 

Third, mixing diverse abilities students in a group might be appropriate for learning by web design 
activity. Through the group collaboration process, the students can learn from each other. In this study, 
many students indicated that their project was successful because of the persons who had good web skills. 
When there was a person who had competence with web skills, other members felt comfortable and learned 
web related skills from the person. A student indicated this as follow. “When I took on the assignment I 
know very little about web design. It sparked interest and now I am much knowledgeable about web 
design. Fortunately we had someone else who did know how to design the page when reduced my anxiety. 
Had that person not been present I may not have undertaken that type of assignment.” 

Lastly, ill-structured content does make a difference. When students search for information from 
various sources and reorganize it, they felt comfortable and felt that they have learned a lot. Based on the 
findings of this research, for the effective use of the web design project, instructors might need to provide 
ill-structured, challenging tasks for the students. To many students, searching and organizing information is 
a challenging task. The learning experiences facilitate their reflective thinking skills. Especially the 
publicity of the web has students more cautious with the content, and students tend to think about the 
content and the web page design in more concrete ways. 



Summary 

According to the constructivist perspective, learning occurs when students pursue individual 
interests and when they engage in authentic activity. From the viewpoint of constructivism, this study 
examined the effects of learning via Web design approach on the perception of the students in a graduate 
course. The findings showed that most students were comfortable and satisfied with the Web design 
activity, and they could learned effectively when they were engaging in both content and design skills. 
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Abstract 

Bias and discrimination based on the differences of others is a serious contemporary problem. 
Biased beliefs often lead to harmful discriminatory action and inhibit emotional and cognitive 
development. Immense human potential is lost by the silencing of difference through hate speech, 
assault, and murder. Such beliefs also serve as perceptual screens that constrict imagination, limit 
experience, and diminish the possibilities of constructing useful meaning regarding difference 
(Greene, 1995). As barriers to new information, varied perspectives, and construction of meaning, 
these perceptual screens are particularly restrictive in educational settings. However, certain 
specifically designed instructional processes may offer significant opportunity for learners to 
develop empathy, reflect on their own biases, and reconstruct stereotypical stories about the 
differences of others. This project explores the use of narrative to mediate the delivery of 
information on the effects of harmful discrimination in a simulated environment that is intended to 
arouse empathy and inspire reflection. It focuses on the potential of instructional narrative 
simulation to change biased beliefs about homosexuality. 



Introduction 



Contemporary theories of psychology and philosophy recognize more than ever the inseparable 
relationship of thoughts and feelings in human mind (Martin & Reigeluth, 1999). Human activity cannot 
be adequately understood without consideration of both affective and rational functioning. Yet, aesthetic, 
emotional, ethical, and social development are all too often ignored in the design and evaluation of 
instructional programs. 

Many recent events, such as the Columbine High School massacre, have sparked painful 
examination of education’s failure to emphasize the more qualitative aspects of human learning and 
development. We are also faced with similarly horrifying residual affects of violent hate crimes in this 
country. For instance, the torture and murder of Matthew Shepard, because he was gay, sparked actions that 
continue to reach beyond the initial horror. Shortly after his murder a college fraternity made light of this 
crime by displaying an effigy of Matthew on their homecoming float A minister and some of his 
parishioners demonstrated at Shepard’s funeral with signs claiming that such punishment was appropriate 

for ho mo sexuals. Even no w, nearly a y ear and a half after his death, a Ch ris tian Internet sit e displays 

animated flames surrounding a photograph of Matthew Shepard while flashing the number of days that he 
continues to bum in hell for the sin of homosexuality.^ 

Biased beliefs about difference, whether related to ability, class, culture, race, or sexual 
orientation, are increasingly problematic as the face of America changes. According to the U.S. Census 
Bureau, non-ethnic European Americans, currently 72% of the population of this country, will decline to 
53% by the year 2050 (January, 2000). Increasing cultural diversity and widespread fear and hatred of 
difference are creating a potentially explosive environment. The prevalence of such attitudes may indicate 
the need for more holistic paradigms of education that include less prescriptive and more innovative 
methods of design, delivery, and evaluation of instruction. 

While affective learning is not new to American education, there has been little focus on actual 
design and evaluation of such instructional products. In a recent review of related resources it was apparent 
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that, while much theoretical work has been done on affective domain learning, little exists in the way of 
instructional strategies and processes for changing attitudes about difference. For example, one curriculum 
resource list includes 153 entries consisting of guides for teachers and administrators, discussion articles, 
pamphlets, activity books, films, recordings, and numerous children’s books, none of which appear to 
include actual strategies for affective learning.^ The Cambridge Friends’ School in Cambridge, MA 
recently circulated an impressive list of children’s books on gay and lesbian families. The Southern Poverty 
and Law Center supports distribution of teacher guides, workbooks, and lesson plans on racism. And 
Rethinking Schools Ltd. distributes teaching materials on issues of equity and social justice.^ While these 
supplemental content materials are important resources, they often lack the instructional design and 
evaluation necessary for changing biased beliefs about difference. 

Project Description 

*'This just is!*' Jeffs Story is an instructional development project that utilizes narrative 
simulation as a strategy for affective learning. While incorporating many resources similar to those 
identified above, it focuses on enhancing empathic abilities, stimulating self-reflection, fueling group 
discussion, and offering opportunity for reconstruction of storied biases. Initially developed as a paper and 
pencil exercise, it has been evaluated through expert review and user testing. Jeffs Story is currently being 
developed into an interactive multimedia program. 

As an instructional strategy, narrative simulation serves to involve the learner as a character in an 
unfolding tale. Jeffs Story was refined from a true story as told by the mother of an adolescent struggling 
with discovering his homosexual orientation."^ The parent and professional version is designed to involve 
the users in the role of Jeffs parents by offering dilemmas, decision points, and answer choices throughout 
the exercise. Learners are faced with difficult decisions that may have serious consequences. These 
decision points are followed by a range of possible answer choices, including helpful answers, biased 
answers, and ones that may be harmful to the adolescent in the story. This decision making process is 
employed to generate self-reflection and imagination. It is also designed to offer users opportunity to 
identify their own biases and reconstruct the storied origins of those beliefs. Interactive links to historical 
accounts, news articles, and resources are included throughout the program to inform users on issues 
related to homosexuality. Sound, imagery, and concern for space, color, and symbolic elements are 
employed as aesthetic mediators to enhance the learning experience. While users are encouraged to focus 
on following the story, they may explore additional information in any sequence they choose, including 
Internet links that are embedded in the program. Design control is concentrated on elements of space and 
environment leaving time and sequence to user discretion. This shift in control-by-design is intended to 
enhance the emotional experience of the user and emphasize the affective goals of this instructional 
composition. 

Conceptualizing the Design of Instruction to Eliminate Bias and Discrimination 

As barriers to new information and varied perspectives, biased beliefs and attitudes inhibit 
emotional and cognitive development by serving as perceptual screens that constrict imagination, limit 
experience, and diminish the possibility of constructing useful meaning regarding difference (Greene, 
■“I995;"Zinchenko,~I997).~Martin and Reigeluth (i 999) echo this perspective in saying that affective leaming 
in the form of emotional growth is both a .foundation for and a component of cognitive development” 

(In Reigeluth, p. 489). However, until recently few affective learning models existed within the field of 
instructional design and technology (Kamradt & Kamradt, 1 999; Martin &. Reigeluth, 1 999). 

Two exciting instructional programs, developed over the past ten years, have inspired and strongly 
influenced the development of Jeffs Story. The Common Thread Case Project is an impressive example of 
instructional design that seeks to inspire independent thinking and develop decision-making skill in pre- 
service and practicing teachers (Bliss & Mazur, 1996; 1998; 2000). Rather than conventional best practice 
models of teacher training, the Common Thread Project offers opportunity for users to consider true story 
cases of teachers in the midst of educational reform. The strategy revolves around consideration and 
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discussion of how users might solve dilemmas presented in the cases. It is an interactive hypermedia design 
that allows users to control time and sequence in navigation. Another important instructional design effort 
that inspired and guided the development of Jeffs Story is a project for use in farm and industrial safety 
programs (Cole, 1993; 1997). Cole relies heavily on cognitive psychology (Bruner, 1986, 1996; Sperry, 
1993), theories of instructional narrative (Fisher, 1995; Howard, 1991; Sarbin, 1986), and research on 
human memory (Neisser, 1978; Schacter & Tulving, 1994) in the development of narrative simulations to 
change attitudes regarding safety practices in mining and farming. While the design of Jeff s Story is 
inspired by the work of these researchers, it is distinctive in focusing specifically on changing biased 
beliefs as a source of discriminatory action. Since such beliefs are socially constructed, the development of 
Jeffs Story also incorporates theoretical perspectives from culture studies (Geertz, 1983; Peshkin, 1991), 
critical social theory (hooks, 1994; Merry, 1990; Yngvesson, 1993), critical race theory (Matsuda, et.al., 
1993), moral education (Appiah & Gutmann, 1996; Ayers, et.al., 1998; Colesante, & Biggs, 1999; 
Kohlberg, 1971; Vitz, 1990), and queer theory (Capper, 1999; Rhoads, 1994; Tierney, 1997). Additionally, 
the storied nature of biased beliefs, as well as particular interest in the instructional qualities of the creative 
arts, provided the basis for also including certain perspectives on human perception and aesthetics (Greene, 
1988; 1995; 1998). 

One of the most useful frameworks for understanding and articulating the design of Jeffs Story is 
a conceptual model for affective development that appears in Volume II of Reigeluth’s Instructional- 
Design Theories and Models , published in 1999. This model includes six dimensions of affective 
development and three identified components of each dimension. The dimensions are emotional, moral, 
social, spiritual, aesthetic, and motivational. Martin and Reigeluth (p. 485-509) identify the components for 
each dimension as knowledge, skills, and attitudes. They stress that the model is incomplete, serving only 
as a beginning in the field of instructional design to address affective development, by including a fourth 
component labeled as other to emphasize that the model is a work in progress. Jeffs Story includes 
mediating elements for most of the six dimensions defined by Martin & Reigeluth. For instance, the 
conceptual model describes the knowledge component of emotional development as “knovdng others 
experience the same emotions you do, such as joy and anger (p. 493).” Jeff s Story employs true story 
narrative in a simulated environment as means to inspire empathic understanding and encourage group 
discussion. 

The instructional model for affective development (Martin & Reigeluth, 1 999) identifies and 
supports a majority of the conceptual foundations for Jeffs Story. However, Jeffs Story includes the 
additional dimension of human activity that appears to be overlooked by Martin and Reigeluth. Activity 
theory offers concepts for describing human activity and includes the central notion of mediation (Nardi, 
1997; Wertsch, 1998). It is concerned with action rather than behavior, relying on concepts of potentiality 
and intent. As such activity theory provides a helpful conceptual frame for articulating the relationship 
between internal mental activity and external material action. 

Empathy, Reflection, and Story Reconstruction as Instructional Outcomes 

As early as 1978 experimental research concluded that in order to promote cooperation among 
interracial groups a dimension of affect must be introduced into the group experience. Cook (1 978) found 
that subjects in a laboratory environment associated positive interracial experiences with exceptional 
individuals instead of interaction with the larger group. Only after some affective dimension was added to 
the cooperative experience did members of the ^oup exhibit empathy for others who were racially 
different. Didactic information and discussion of differences alone had no impact on the development of 
empathic understanding. This study is important for the design of instruction to affect biased beliefs 
because it emphasizes the interrelationship of affective mediators, empathic understanding, and 
cooperation. It also demonstrates the feasibility of creating instructional environments that are analogous to 
real world learning settings. ''This just w!” Jeffs Story employs true story narrative and simulation design 
to create experiences that are analogous to those in the real world. It involves learners in actual dilemmas 
and decisions to increase the likelihood that users will identify personally with the story. 

Beliefs are constructed in memory as storied truths. Reconstruction of beliefs requires a process of 
reflection. Jeffs Story is designed to engender reflection through dilemmas that users are asked to resolve 
by choosing certain answers among a range of answer choices. Once an answer is chosen a new screen 
appears that offers a discussion of that answer choice. Continuing on with the narrative, users discover how 
Jeffs parents responded and the resulting consequences. In order to further stimulate reflection the exercise 
is followed by a group discussion focused on comparison and explanation of answer choices. This process 
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of comparing one’s own responses to those of other users is intended to encourage learners to examine their 
own beliefs. Discussion of varying responses among members of the group is crucial in providing a 
generative experience that may eventually lead to examination of familial and social origins of biased 
beliefs. 

Human beings make meaning of the events in their lives by associating those events with existing 
understandings. These schemas are dominant mechanisms for restructuring memories and recall is distorted 
to fit existing schema (Anderson, et.al., 1996). The process of making meaning is an activity that is limited 
by previously associated meanings held in one's mind (Polkinghome, 1 988). Jeffs Story provides 
dissonance and contradiction by creating space for discussion of varying opinions on emotionally charged 
topics. Such dissonance is an internal imbalance that causes learners to seek new understandings as a way 
to restore cognitive equilibrium (Piaget, 1970). These newly acquired understandings can be seen as objects 
of expanded potential in individual mental processing (Zinchenko, 1997). An even more disturbing aspect 
of this simulation is Jeffs suicide. Despite all their efforts to support and encourage him, Jeffs parents 
were unable to prevent his death. Learners are left to struggle with what they might have done differently. 
Such a disturbing end offers potential to extend reflection beyond actual instructional time and space. In 
other words, the instructional experience sets the stage for new possibilities. This potential is a single entity 
of time and space that serves as a means to new activity (Zinchenko, 1997). 



Proof of Concept 

Jeffs Story has been field tested with a variety of users in both the original pencil and paper 
format and as an interactive multimedia program. Prior to user testing the exercise was reviewed by a 
variety of experts who made editorial suggestions. This advice generally focused on changes in the wording 
of questions, adding answer choices, and revising some of the didactic information. After working the 
exercise for the first time, several of these experts reacted with strong emotional statements. They were 
upset that the story ended with Jeffs suicide or thought that the exercise could be hurtful to certain users. 
Some of the most thoughtful suggestions came from the woman who wrote the original story of her own 
son’s suicide. She was able to provide feedback to assure the fidelity and veracity of the distilled narrative. 
Revisions based on these reviews were incorporated before the paper and pencil exercise was tested with 
potential users. 

Initial field-testing was completed with fourteen adults, including parents, ministers, and married 
or single professionals who had no children of their own. The objective was to obtain information on: a) 
veracity, b) usability, c) group discussion, and c) relevance of the didactic information presented in the 
exercise. Evaluations were completed immediately following the exercise and several weeks later. The 
results reflected that 86% felt the story was real, 93% felt the instructions were easy to follow, and the same 
percentage said it was useful to hear viewpoints of others during the group discussion. Additionally, 71% 
felt the group discussion caused them to rethink some of their own answers and almost half indicated that 
the questions were difficult for them to answer. These results reinforced the importance of maintaining 
group discussion as an integral part of the learning process. 

Five user tests have been completed on the multimedia version of Jeffs Story. The concern in 
these tests was to obtain user feedback regarding navigational links, general usability, and average time to 
work through the simulation. These users included four females and one male, ranging in age from 33 years 
to 55 years. Straight, gay and bisexual identities were represented and the group included two parents. 
Computer skills ranged from novice to expert and it took an average of 25 minutes to complete the 
simulation. The most helpful feedback here was related to navigation, affective reactions, and content 
suggestions. First, as expected, many internal and Internet links did not function properly. For example, one 
user pointed out that the first two screens indicated the main link would be found centered at the bottom of 
the screen but it was located on the global navigation bar on the left of the screen throughout the rest of the 
program. This and all other navigational problems were quite easy to correct. While such feedback was 
extremely important in the development process, the user comments regarding the overall program were 
enlightening and reiterated much of the user feedback on the paper and pencil version. Most of the negative 
responses revolved around the answer choices. Users indicated they had difficulty choosing only one 
answer and some felt it was not clear enough in the instructions that they could choose as many answers as 
they wished. One user didn’t like any of the answer choices for two of the questions and offered 
suggestions for additional responses to those dilemmas. These users felt generally positive about the 
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experience and several indicated the experience provoked reflection and discussion with friends and family 
members. One user had a very strong emotional reaction to the suicide. She said it surprised her and that it 
could have been prevented. 

The most important information gained from user feedback is perhaps the indication that that the 
experience stimulated continued thinking and discussion. Since reflection is a necessary activity for 
reconstructing biased beliefs, such feedback offers hope that this instructional experience, at least, can 
create the possibility of changing biased beliefs over time. In fact, most of the information collected thus 
far reinforces the theoretical foundations on which this project is built. 



Summary 

Can instructional narrative stimulate open discussion of homophobia? Will such discussion cause 
significant reflection on the nature and origin of negative beliefs about homosexuality? To what extent will 
such reflection result in the reconstruction of stereotypical stories about difference? ''This just w!” Jeff’s 
Story is unique as an instructional product that combines affective learning with contemporary social 
problems in an attempt to answer such questions. 

Although much research has been done in social psychology and human learning, the application 
of such discourse to the design and evaluation of educational products that are specifically focused on 
biased beliefs about difference is rare. As early as 1954, Gordon Allport called for “militant tolerance” or 
the need for citizens to become intolerant of intolerance. Yet, in his historical study of the Holocaust, 
Browning (1992) concluded that the urge to conform was the underpinning that allowed 500 working class 
German men, with no particular interest in Nazi dogma, to murder nearly 40,000 Polish Jews and send 
another 45,000 to their deaths in Treblinka. 

In the United States legal remedies, educational policy, behavioral approaches, and didactic 
curriculum materials seem to have little impact on attitudes about difference. The urge to conform is 
reiterated in numerous accounts of the torture and murder of homosexuals and African Americans. While 
mere tolerance of difference may be possible through behavioral approaches, legal mandates, and 
educational policies, independent thinking and valuing difference require mental activities such as 
reflection and empathy. Such mental activities empower and enable militant tolerance and dialogue about 
difference. 

This project focuses on discriminatory action as rooted in biased belief, biased belief as socially 
constructed in narrative form, and narrative as an aesthetic mediator to promote critical thinking and 
affective change. It uses narrative simulation to enhance perception of the present and stimulate 
imagination of possibilities and potential (Greene, 1995). Human beings have both the necessity and 
potential to embrace and enjoy differences. Such potential can be mediated through instructional tools that 
stimulate new mental activity. 

This instructional program is aimed at homophobia because fear and hatred of people who are gay 
remains largely condoned and openly expressed in contemporary society. While biased beliefs of a racist or 
sexist nature are also prevalent, open expression of those is widely discouraged. The urge to conform to 
social norms often results in much more subtle expressions of racist or sexist biases. Therefore, initial focus 
on homophobia appears to have more potential to engender open discussion and reflection. The vision, 
however, is to gain insight through the development and evaluation of Jeffs Story that will assist in 
designing instructional programs focused on a range of biased beliefs. 
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Applying Social Cognitive Constructs of Motivation 
TO Enhance Student Success in Online Distance 

Education 



This paper relates findings from a review of the literature on six motivational constructs to 
instruction in face-to-face and distance education environments. The six motivational constructs, 
which are founded on social cognitive learning theory, are self efficacy, locus of control, 
attributions, goal orientation, intrinsic versus extrinsic motivation, and self regulation. These 
motivational constructs have been thoroughly investigated in traditional classrooms but few 
studies have explored their significance in the context of distance education. The purpose of this 
paper is to relate the motivational constructs studied in traditional environments to distance 
education concepts in order to identify methods for ensuring student success in online courses. 

Introduction 

This paper relates findings from a review of the literature on six motivational constructs, founded 
on social cognitive learning theory, to instruction in face-to-face and distance education environments. The 
six motivational constructs are self-efficacy, locus of control, attributions, goal orientation, intrinsic versus 
extrinsic motivation, and self-regulation. They are divided into three general families according to the type 
of cognitive structures they influence. The three categories are (a) individuals' perceptions about their 
ability to accomplish a task (self-efficacy, locus of control, and attributions), (b) individuals' reasons or 
purposes for engaging in a task (goal orientation and intrinsic versus extrinsic motivation), and (c) 
individuals' techniques and strategies for accomplishing a task (self-regulation). The six motivational 
constructs have been thoroughly investigated in traditional face-to-face classrooms and in a wide variety of 
educational disciplines. Very few studies, however, have explored the significance of the constructs in the 
context of distance education. The purpose of this paper is to relate the motivational constructs studied in 
traditional environments to distance education concepts in order to identify methods for ensuring student 
success in web-based courses. 

Distance Education 

Historically, distance education has been defined as the delivery of instruction in which time and 
geographic location separate students and teachers (Mclsaac & Gunawardena, 1996). Distance education 
courses have been an alternative for students in remote areas, or with difficult time schedules, unable to 
participate in traditional classroom instruction. The lack of face-to-face meetings between teachers and 
students in a shared classroom has led to the development of systems of delivering instruction in modalities 
often very different from those used in traditional teaching. Over the years different delivery systems have 
evolved along with the development of communication technology. In the late 1800s, traditional mail 
played the role of the first delivery system in correspondence studies. During the next hundred years, radio, 
television, and computer multimedia were used to deliver instruction. The development of advanced 
technologies in the last decade has had an important impact on education. As a result, computer-mediated 
communication (CMC) is becoming the leading way to reach distance learners and proving to be a global 
communication system (Harasim, 1 996). 
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The present trend and market demand for education is indeed toward distance education. This 
trend is highlighted in the "The Condition of Education" report published by the National Center for 
Education Statistics (NCES, 1999), According to this report, in 1995 thirty-three percent of higher 
education institutions offered distance education courses and another twenty-five percent indicated plans to 
begin courses within three years. Furthermore, Peterson's (1999) Distance Learning Guide provides 
information about two thousand degree and certificate programs available from nearly 900 institutions. 

This number can be compared to 762 institutions in 1997 and 93 institutions in 1993. Many educational 
institutions offer a wide variety of online courses and provide the opportunity for students to enroll in 
certain online courses as part of a degree. Other institutions offer complete undergraduate and graduate 
degrees through online distance education. Finally, the Kellogg Commission on the Future of State and 
Land-Grand Universities (1999) recently published a report entitled "Returning to Our Roots: A Learning 
Society" Members of the commission include presidents of state institutions that have invested heavily in 
technology in recent years through statewide distance education programs. According to the report, online 
technology needed for universal access to education is available today. 

Technology, interactivity, and attrition rates are important issues in the area of distance education. 
The following sections describe in detail the three issues that are related to the concept of motivation. 

Technology and Computer-mediated Communication 

While technology in general is the backbone of online environments, CMC is the gateway for 
thousands of online learners in virtual communities. CMC refers to the use of networked computers for 
communication, interaction, and exchange of information between students and instructors (Berge & 
Collins, 1995), Examples of CMC include electronic mail (Email), bulletin boards, newsgroups, and 
computer conferencing. The rapid growth of computer networks and the evolution of the Internet in the last 
decade have magnified the use of CMC to the point that it now plays an essential role in web-based 
delivery of instruction. Studies have shown that online learners interact with their peers, instructors, and 
content experts in ways that allow students to develop their critical and problem solving skills (Riel, 1993), 

The use of communications technology is still a recent development in education however, and 
many online students encounter various difficulties with using such technologies. Novice students, for 
example, tend to feel apprehensive about using computer hardware, CMC, and the Internet in ways that 
may jeopardize intellectual interaction and their ability to succeed in a web-based course. Moreover, 
students who are technologically illiterate tend to spend many hours trying to figure out how to use online 
technologies, communicate with instructors, submit online assignments, or download class-related material 
from the class web site. Thus additional research may be needed in order to determine students’ capabilities 
with using online technologies in order to enable instructors to provide immediate remediation to students 
early in the semester and therefore increase interaction. 

Interaction 

Interaction in an online environment is an important factor that can influence the success or failure 
of a course. Keegan (1988) views interaction as a key to effective learning and information exchange, and 
Moore (1989) considers interaction as very important in the design of distance education. Furthermore, 
Kearsley (1995) points out that a high level of interaction positively influences the effectiveness of any 
distance learning course. 

There are four types of online interaction identified in the literature. Moore (1989) identifies 
learner-content, learner-instructor, and learner-learner interaction. Hillman, Willis, and Gunawardena 
(1994) identify a fourth type of interaction, namely learner-interface interaction. 

Moore (1989) defines learner-content interaction as the intellectual interaction between the learner 
and the topic of study. According to Moore, learner-content interaction is an important concept of online 
environments because it changes learners’ behavior toward an educational goal. The importance of this 
type of interaction is also depicted by Moore and Kearsley (1996) in the statement: "it is not too difficult to 
present information over a distance, but getting people to participate and making learning active at a 
distance is much harder" (p. 133). 

The second type of interaction in online environments occurs between learners and instructors or 
team of subject-experts who prepared the course material. In this type of interaction, instructors are 
responsible for stimulating and continuously maintaining learners’ interest in the topic, motivating students 
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to leam, assessing students’ progress, and finally providing support and encouragement to them (Moore, 
1989). 

The third type of interaction occurs among the learners of an online environment with or without 
the real-time presence of instructors. This type of interaction represents the communication between one 
learner with another learner, or with a group of learners, and takes place either synchronously, via "live" 
discussion chats, or asynchronously, via the exchange of electronic email or posting of messages in bulletin 
boards. 

Hillman et al. (1994) define learner-interface interaction as "a process of manipulating tools to 
accomplish a task" (p. 34). Learners must understand not only the procedures for working with the 
interface, but also the reasons why these procedures obtain results. This fourth type of interaction links the 
other three types of interaction together. Learners must be able to use online technologies in order to 
interact and communicate with instructors, peers, and the course content. The absence of interaction can 
inhibit student success and may even force online students to drop out of online courses. 

Attrition 

Attrition rates in distance education courses can be forty to fifty percent higher than the ones in 
traditional face-to-face classrooms (Dille & Mezack, 1991; Parker, 1994). Distance education requires 
students to monitor and regulate their own learning because of the geographic separation between students 
and instructors. Inquiry that sheds light on online students’ motivational characteristics and organizational 
skills is vital in order to empower educators to design instructionally sound courses and 
students to benefit from them. Such research studies would aid academic administrators 
predict student success with the ultimate possible purpose of lowering attrition rates. 

Motivation 

Motivation is one of the most important components of learning in any educational environment. 
Questions about why students engage in, pursue, and accomplish certain goals or tasks, or why they avoid 
others, have been the subject of scholarly inquiry since the writings of fifth-century BC Greek 
philosophers. There are many definitions of motivation that have emerged from various theoretical 
approaches during the last half of the twentieth century. In general, motivation (a) increases individuals’ 
energy and activity levels (Maehr, 1984), (b) directs individuals toward certain goals (Dweck & Elliot, 
1983), (c) promotes initiation of certain activities and persistence in those activities (Stipek, 1988), and (d) 
affects the learning strategies and cognitive processes individuals employ (Dweck & Elliot, 1983; Eccles & 
Wigfield, 1985). 

Social cognitive learning theory views motivation as a function of individuals’ thoughts rather 
than some instinct, need, drive, or incentive as examined by Freud (1915), Hull (1943), and Maslow 
(1954). Through the lens of social cognitive learning theory, six motivational constructs can be classified 
into three general categories: (a) individuals' perceptions about their ability to accomplish a task, (b) 
individuals' reasons or purposes for engaging in a task, and (c) individuals' techniques and strategies for 
accomplishing a task. The relevant constructs for examining students' perceptions are self-efficacy, locus of 
control, and attributions. Students' reasons for engaging in a task can be examined by looking at their 
achievement goal orientation and the intrinsic versus the extrinsic influences. Students' techniques and 
strategies for accomplishing a task can be explored by looking at self-regulation. Each of the three 
categories of motivational constructs will be examined below. 

Individuals’ Perceptions about their Ability to Accomplish a Task 

Perceptions of ability play an important role in all cognitive theories of achievement motivation. 
The basic idea behind this family of motivational constructs revolves around students' beliefs that they are 
able to perform a task and that they are responsible for their own performance. These motivational 
constructs answer the student’s question: "Can I do this task?" 

In Bandura's Social Cognitive Theory (1986), individuals who have a high perception of self- 
efficacy for accomplishing a task work harder and persist longer when they encounter difficulties than 
those who do not feel efficacious. In Rotter's Social Learning Theory (1966), individuals who believe they 
are academically competent are more likely to believe they control academic outcomes. Similarly, in 
Weiner's Attribution Theory (1985, 1986), individuals who believe they are competent at a task will 
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probably attribute their success to their ability; in contrast, they will probably attribute their failure to some 
other external cause. 

Self-efficacy 

Self-efficacy is a major component of Bandura's (1986) social cognitive learning theory. Bandura 
(1986) describes self-efficacy as individuals' confidence in their ability to control their thoughts, feelings, 
and actions, and therefore influence an outcome. These perceptions of self-efficacy influence individuals’ 
thoughts, emotions, choices of behavior, and amount of effort and perseverance expended on an activity. 
According to Bandura (1986), individuals make personal ability judgments and evaluations through a 
cognitive appraisal system that is unique to the individual, the task, and the particular situation at any given 
moment. Because self-efficacy judgments are task-specific, Bandura (1986) cautions that a self-efficacy 
measure must assess the specific skills needed for performing an activity, and it must be administered 
during the time that the performance is being assessed. 

According to Bandura (1986), individuals acquire information to help them assess self-efficacy 
from four principal sources: (a) actual experiences, (b) vicarious experiences, (c) verbal persuasion, and (d) 
physiological indexes. 

Individuals' own performances, especially past successes and failures, offer the most reliable 
source for assessing efficacy. Typically, successes raise efficacy appraisals and failures lower them. 
However, once individuals develop a strong sense of efficacy, a failure may not have much impact 
(Bandura, 1986). 

Vicarious experiences can also affect self-perceptions of efficacy. According to Schunk (1989b), 
peers offer the best basis for comparison. Observation of similar peers performing a task conveys to 
observers that they too are capable of accomplishing that task. In a study by Schunk & Hanson (1985), 
children who were having difficulty with subtraction problems observed either a same-sex peer or a teacher 
demonstrate mastery. Results of the study show that children who observe a peer succeeding at a task have 
a higher self-efficacy for learning the procedure as opposed to those who observe teachers. Information 
acquired vicariously typically has a weaker effect on self-efficacy than does personal experience-based 
information (Bandura, 1986). Bandura also indicates that a vicarious increase in self-efficacy could be 
negated by subsequent failures. 

Verbal persuasion is the third factor that influences self-efficacy judgments. Persuasion is when 
individuals are encouraged to believe that they possess the capabilities to perform a task (e.g. being told 
"you can do this"). Verbal persuasion is not likely to be effective unless it is realistic and reinforced by real 
experience. Furthermore, verbal persuasion enhances self-efficacy, but this increase will be temporary if 
subsequent efforts result in failure (Bandura, 1986). 

The last factor influencing self-efficacy judgments are physiological indexes. Individuals acquire 
efficacy information from physiological states such as heart rate and sweating. Bodily symptoms signaling 
anxiety and fear might be interpreted by the students themselves to indicate their own lack of skills. 

Information acquired from the above sources does not necessarily influence self-efficacy. 
According to Bandura (1986), information gained through these sources is cognitively appraised before an 
efficacy judgment is made. Efficacy evaluation is an inferential process, in which individuals weigh and 
combine the contributions of personal and situational factors. These factors are: (a) their perceived ability, 
(b) the difficulty of the task, (c) the amount of effort expended, (d) the amount of external assistance 
received, (e) the number and pattern of successes and failures, (f) their perceived similarity to models, and 
(g) the persuader credibility (Schunk, 1989b). 

Researchers in academic environments have studied the relationship among self-efficacy and 
constructs such as goal orientation (Anderman & Midgley, 1992; Urdan, Pajares, & Lapin, 1997; 
Zimmerman, Bandura, & Martinez-Pons, 1992) and self-regulation (Bandura, 1993; Pintrich & De Groot, 
1990; Pintrich & Garcia, 1991; Schunk, 1991). Results indicate that students who believe they are capable 
of performing certain tasks use more cognitive skills and metacognitive strategies and persist longer than 
do those who do not feel they are capable of performing certain tasks (Pintrich & Garcia, 1991), Pintrich 
and De Groot (1990) report that academic self-efficacy correlates with academic outcomes such as exam 
scores and final year grades. In a similar manner, Schunk (1991) states that individuals who have a high 
sense of self-efficacy for accomplishing a task work harder and persist longer when they encounter 
difficulties, whereas those who do not feel efficacious may quit or avoid a task. In the same context, 
Bandura (1993) states that when individuals with high self-efficacy are challenged by a difficult situation 
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they are more likely to attempt different strategies, or develop new ones, and are less likely to give up than 
people with a low sense of self-efficacy. 

Individuals who feel they lack the ability to affect an outcome may act in ways that prevent 
learning and performance. For example, individuals with low self-efficacy beliefs tend to avoid activities 
they believe are beyond their capabilities and therefore, choose easier tasks where the chances for success 
are greater. Furthermore, individuals with low self-efficacy do not try to develop new skills and strategies, 
as they feel uncertain about their previous skills (Bandura, 1 986), These individuals invest less effort in an 
outcome and give up more quickly than do those with high self-efficacy. This poor performance reinforces 
low self-efficacy, which leads to poorer performance, and so on. Some researchers call this "learned 
helplessness*' (Bandura, 1982, 1986, 1993; Peterson, Maier, & Seligman, 1993), Goal orientation theorists 
refer to learned helplessness as a "helpless response" and state that these responses are the result of high 
performance goal orientation in addition to low efficacy (Dweck, 1986; Dweck & Leggett, 1988; Nicholls, 
1984, 1989). 

Researchers have established that self-efficacy is a strong predictor of academic performance in 
traditional face-to-face classrooms. Multon, Brown, and Lent (1991) reviewed a comprehensive list of 
studies that examined self-efficacy in achievement situations. Findings suggest that self-efficacy beliefs are 
positively related to academic performance. In the same context, Ames (1984) and Nicholls and Miller 
(1994) suggest that students' self-perceptions of ability are positively related to achievement and student 
motivation. 

Locus of control 

Another construct influencing students* perceptions of ability is locus of control. Locus of control 
is a relatively stable trait, a belief about the extent to which behaviors influence successes or failures 
(Rotter, 1966). It affects learning, motivation, and behavior (Pintrich & Schunk, 1996), Individuals with an 
internal locus of control believe that success or failure is due to their own efforts or abilities. On the other 
hand, individuals with an external locus of control are more likely to believe that other factors, such as 
luck, task difficulty, or other people's actions, cause success or failure. 

Locus of control is an important factor in explaining students* school performance. Several 
research studies show that students who have a high internal locus of control earn better grades and test 
scores than do students of the same intelligence who have a low internal locus of control (Schunk, 1991; 
Shell, Colvin, & Brunning, 1 995). Other studies also show that locus of control is an important predictor of 
student academic achievement (Pajares & Miller, 1994; Randhawa, Beamer, & Lundberg, 1993; 
Zimmerman & Bandura, 1994; Zimmerman et al., 1 992), In the distance education context, several studies 
indicate that students with an internal locus of control are more likely to be successful in a distance learning 
course (Dille & Mezack, 1991; Mclsaac & Gunawardena, 1996; Parker, 1994). 

Attributions 

Causal attributions are individuals’ perceptions of the causes of various achievement outcomes 
(Pintrich & Schunk, 1996). Students may attribute their successes or failures to two general types of 
antecedent conditions: environmental factors and personal factors. Environmental factors include teacher 
feedback, social norms, or situational features. Personal factors include causal patterns, personal bias, prior 
knowledge, or individual differences. These two general categories of perceived causes influence the actual 
attributions that individuals will make in terms of whether they attribute their failure to low ability, lack of 
effort, bad luck, a hard test, a bad mood, fatigue, unfairness, anxiety, or just about any other explanation, 
justification, or excuse students produce for failure at a test or task. 

According to Weiner (1986), most of the causes to which students attribute their successes or 
failures can be characterized in terms of three dimensions: locus of causality, stability, and control. The 
locus-of-causality dimension is similar to Rotter's (1966) locus of control construct, and refers to whether a 
cause is perceived as being internal or external to the individual. For example, ability and effort are both 
classified as internal causes, whereas task difficulty and luck are classified as external causes. The stability 
dimension refers to whether the cause is fixed and stable (e.g. aptitude and task characteristics), or whether 
it is variable and unstable across situations and over time (e.g. skills, knowledge, and chance). The 
controllability dimension refers to how much control a person has over a cause. 
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The three constructs described above fall into the first general category of motivational constructs, 
which represents students' perceptions about their ability to accomplish a task. The following section 
describes the second general category of motivational constructs. 

Individuals’ Reasons or Purposes for Engaging in a Task 

Individuals' reasons or purposes for engaging in a task also play an important role in cognitive 
theories of achievement motivation. The basic idea behind this family of motivational constructs concerns 
students' incentives for engaging in a particular task. This family of constructs answers the student’s 
question: "Why am I doing this task?" 

Goal orientation 

An achievement goal is what an individual is striving to accomplish (Locke & Latham, 1990). 
According to Locke and Latham ( 1 990), there are four main reasons why goal setting improves 
performance: (a) goals direct students’ attention to the particular task, (b) they engage effort, (c) they 
increase persistence, and (d) they promote the development of new strategies when old strategies fail. 

Dweck ( 1 986) and Dweck and Leggett ( 1 988) have identified two motivational patterns that are 
associated with differences in individuals' goal orientation. The first is the "mastery response," associated 
with learning goal orientation (Elliot & Dweck, 1988), in which challenging tasks are sought and effort is 
increased in the face of difficulty. Learning goals have also been called mastery goals (Ames & Archer, 
1988), task incentives (Maehr & Braskamp, 1986), and task involvement (Nicholls, Patashnick, & Nolen, 

1985) . 

Performance goal orientation is associated with the "helpless response" pattern (Ames & Archer, 
1988; Elliot & Dweck, 1988), in which challenging tasks are avoided and performance decreases when 
difficulty is encountered. Performance goals have also been called ego incentive (Maehr & Braskamp, 

1986) , or ego involvement (Nicholls, Patashnick, & Nolen, 1985). 

Research on the relationship between learning and performance goals indicates that these two 
types of goals are independent of one another (Hagen & Weistein, 1 995), rather than opposite of one 
another as suggested in early motivational research (Meece & Holt, 1993). Pintrich and Garcia (1991) 
suggested that this independence means that it is possible for students to have both learning and 
performance goals at the same time. 

A review of the literature on achievement goal orientation reveals that learning goals are positively 
related to self-efficacy (Hagen & Weistein, 1995; Urdan et al., 1997), and self-regulation (Ames & Archer, 
1988; Hagen & Weistein, 1995; Meece, Blumenfeld, & Hoyle, 1988; Nolen, 1988). For example, in a study 
by Schunk (1995), results indicate that learning goal orientation leads to higher self-efficacy and self- 
regulated performance than performance goal orientation. Individuals with a learning goal orientation strive 
to master a particular task and to improve themselves no matter how many mistakes they make. Their 
primary goal is to obtain knowledge and improve their skills. Consequently, they may process information 
at a deep level (Miller, Behrens, Greene, & Newman, 1993; Nolen, 1988). These individuals extend their 
learning processes beyond the minimum required and pursue the learning process as long as they perceive 
that they are making progress. The combination of these factors enables leaming-goal-oriented individuals 
to learn more and to perform better than individuals motivated only to perform better than others. 

Learning goal-oriented students are more likely to engage in self-regulatory activities such as the 
use of monitoring, planning, and deep-level cognitive strategies (Ames & Archer, 1988; Anderman, 1992; 
Graham & Golan, 1991; Meece, Blumenfeld, & Hoyle, 1988; Nolen, 1988). Students who adopt learning 
goals also tend to find the topic under study more intrinsically rewarding (Meece et al., 1 988; Miller et al., 
1993; Nicholls & Miller, 1994). Furthermore, students with a learning-goal orientation tend to achieve 
higher on tasks and to persist longer after failure, compared to performance-goal-oriented students (Diener 
& Dweck, 1978; Dweck & Leggett, 1988). 

Individuals oriented toward performance goals are concerned with positive evaluations of their 
abilities in comparison to others. They are focused on how they are judged by others (such as peers, 
teachers, or parents). They want to look smart, and they try not to seem incompetent. For these reasons, 
they may avoid challenging tasks and exhibit low persistence when they encounter difficult work (Ames & 
Archer, 1988; Dweck & Leggett, 1988; Elliot & Dweck, 1988; Maehr & Midgley, 1991; Nicholls, 1989). 
By doing so, they adopt failure-avoiding strategies such as pretending not to care, making a show of "not 
really trying", or simply giving up (Jagacinski & Nicholls, 1987; Pintrich & Schunk, 1996). The evaluation 
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of their performance is what matters to them, instead of learning the course material. Individuals with a 
performance-goal orientation tend to process information at a superficial level and generally fail to pursue 
learning beyond the level necessary to achieve positive recognition. Consequently, they frequently fail to 
retain the information they learn (Greene & Miller, 1996; Miller et al., 1993; Nolen, 1988; Pintrich & 
Garcia, 1991). 

Intrinsic and extrinsic motivation 

Students’ reasons or purposes for engaging in tasks are also influenced by their relative intrinsic 
and extrinsic motivation. Motivation that stems from factors such as interest or curiosity is called intrinsic 
motivation. Intrinsic motivation is the natural tendency to seek out and conquer challenges as individuals 
pursue personal interests and exercise their capabilities (Deci & Ryan, 1985). When students are 
intrinsically motivated, they tend not to need any incentives because the activity itself is rewarding to them. 
In contrast, extrinsic motivation is motivation to engage in an activity as a means to an end. Students who 
are extrinsically motivated tend to work on tasks because they believe that participation will result in 
desirable outcomes such as a reward (a good grade, or a diploma), teacher praise, or avoidance of 
punishment. Pintrich and Schunk (1996) suggest that intrinsic motivation and extrinsic motivation represent 
two different continua and each range from high to low. 

Goal orientation and intrinsic versus extrinsic motivation fall into the second general 
category of motivational constructs, which examines students’ reasons or purposes for engaging in a task. 
The next section describes the third category of motivational constructs. 

Individuals’ Techniques and Strategies for Accomplishing a Task 

Individuals' techniques and strategies for accomplishing a task also play an important role in 
cognitive theories of achievement motivation. The basic idea behind this motivational construct concerns 
students’ utilization of cognitive skills and metacognitive strategies in order to accomplish a task. This 
construct answers the student’s question: "How can I do this task?" 

Self-regulation 

Self-regulation refers to students' ability to understand and control their learning (Schunk & 
Zimmerman, 1 994; Zimmerman, 1 994). Self-regulated learning is comprised of two components, (a) 
learning strategies, which include cognition and metacognition, and (b) motivation. Self-regulated students 
who employ cognition and metacognition plan, organize, self-instruct, and self-evaluate at various stages 
during the process of information acquisition. Students who are motivated perceive themselves as self- 
efficacious, intrinsically motivated, and goal-directed (Zimmerman, 1989). As a result, self-regulated 
students are often academically superior to students who do not self-regulate their learning experiences 
(Pintrich & De Groot, 1990; Zimmerman, 1986; Zimmerman & Martin ez-Pons, 1986, 1988). 

Zimmerman (1994) identifies four attributes of self-regulated learning: (a) self-motivation, (b) 
self-monitoring, (c) manipulation of the social and physical environment, and (d) self-confidence. Self- 
motivation refers to motivation that is derived from students' self-efficacy perceptions and their use of self- 
regulatory learning processes, such as setting goals. Self-monitoring refers to students’ awareness and self- 
checking during a learning process. Manipulation of the social and physical environment refers to students’ 
ability to both seek help from people who they know are capable, and also organize and restructure their 
skills in order to optimize learning. Self-confidence refers to the planned or automated methods of learning. 
O’ Neil (1978) calls these methods of learning learning strategies. Weinstein & Mayer (1986) classified 
them into two major categories: (a) learning strategies associated with outcome goals, and (b) learning 
strategies associated with process goals (such as monitoring, controlling, planning, organizing, 
transforming, rehearsing, and memorizing). Zimmerman and Martinez-Pons (1986) have defined the latter 
category of learning strategies as self-regulation. Pintrich and De Groot (1990) included several of these 
strategies in their Self-Regulated Learning Strategies scale. This is one of the two parts that compose the 
Motivated Strategies for Learning Questionnaire (MSLQ) (Pintrich, Smith, Garcia, & McKeachie, 1991). 

A review of the literature uncovered numerous theoretical and empirical studies on self-regulation 
(Garcia, 1995; Pintrich & Garcia, 1991; Schunk & Zimmerman, 1994). Garcia (1995) proposes that 
students use their self-efficacy beliefs to fuel their motivational strategies. Pintrich and De Groot (1990) 
suggest that increased levels of self-efficacy stimulate self-regulated learning. Meece (1994) indicates that 
self- regulated learners possess motivational attributes in their goal orientation that affect their learning 
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experiences. For example, some students are intrinsically motivated to engage in academic activities, while 
others are extrinsically motivated to maintain their engagement. In another study, Malpass, O’Neil, and 
Hocevar (1999) investigate the effects of self-regulation, goal orientation, self-efficacy and high-stakes 
math achievement for mathematically gifted high school students. Results indicate that self-efficacy is 
positively related to math achievement and self-regulation, and that learning goal orientation is positively 
related to self-regulation. 

The evidence presented in the above studies points towards the importance of self-regulation and 
its components as a predictor of academic achievement in traditional face-to-face classrooms. One area of 
study concerning self-regulation that has not yet been completely examined is that of its effects on students' 
achievement and satisfaction in online courses, as well as on course completion. 

In light of high attrition rates within some distance education environments and increasing number 
of courses utilizing online learning support through Internet delivery, even traditional students are likely to 
experience distance education methods in the near future. Providing information that reveals learners’ 
motivation and strategies for successful engagement in distance learning environments is likely to prove 
beneficial for both distance education and traditional education that is supported through online education 
methods. 

Implications 

Given the present body of literature, it is evident that there are implications for (a) 
research on students’ actual CMC skills, as well as research on students’ self-efficacy 
beliefs with CMC, and (b) instructional design and course development principles for 
online courses. 

Research Implications 

The literature review on the three motivational constructs in traditional and distance education 
environments has uncovered the need for more research in the online environment. Specifically, future 
research should examine the role of motivational constructs in online distance education in order to identify 
the motivational characteristics of online students and predict their success. 

Several researchers have identified the importance of investigating the joint effects of self- 
efficacy, goal orientation, and self-regulation on academic achievement and performance in traditional 
environments (Kane, Marks, Zaccaro, & Blair, 1996; Malpass, 1994; Malpass et al., 1996; Malpass, O'Neil, 
& Hocevar, 1999; Yap, 1993). Kane et al. (1996) found that personal goals and self-efficacy beliefs 
positively influenced the performance of 216 wrestlers competing in a wrestling camp. Malpass (1994), 
Malpass et al. (1996), Malpass et al. (1999), and Yap (1993) found that self-efficacy is positively related to 
math achievement and self-regulation, and that learning goal orientation is positively related to self- 
regulation. Future research should investigate the role of self-efficacy, goal orientation, and self-regulation 
in the online environment in order to predict student success in online courses. Each construct represents 
one of the three general categories discussed previously and all three may play a vital role in an equation to 
predict student success in online courses. 

Research on students’ actual CMC skills 

Distance education theorists in the early 1980's such as Boije Holmberg (1980) and David Sewart 
(1981) advocated the necessity for effective student support systems in order to prevent students from 
dropping out of distance education courses. Holmberg’s guided didactic conversation theory (1980) 
described the relationship between distance education students and their support systems such as 
correspondence and telephone communication. Likewise, Sewart's (1981) theory of intermediary stated that 
distance education students have individual needs and only by providing someone in between the student 
and the system will those needs be fulfilled and students will succeed. For example, Sewart (1981) 
suggested appointing counselors in order to both facilitate learning and help motivate students. Similar 
ideas could be adopted in the context of the online environment. The following sections describe 
suggestions based on Holmberg’s and Sewart’s theories of distance education for helping online students 
persist in an online course. 
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Students enrolled in online courses are often required to use technology for the daily procedures of 
the course. Such procedures include (a) interacting with peers and instructors via email, (b) using a web 
browser to access class material, (c) searching for journal articles using the Internet, online databases, and 
the institution's libraries, (d) submitting assignments online, and (e) participating in weekly asynchronous 
threaded discussions. These are examples of the four online types of interaction. Moore (1989) suggested 
that online students need to interact with the instructor, the course content, and their peers. The only way 
for any interaction to take place is for students to be able to use online technologies such as the Internet, 
email, and CMC systems. All four types of interaction are vital to online environments because if one type 
of interaction is missing, or it is not well thought and planned, then online courses might not be effective 
and successful. For example, if there is no interaction between learners and the topic being studied (because 
of unspecified or unclear instructions, assignment questions, or discussion topics), then learners will not 
obtain the desired learning outcomes. Moreover, if learners cannot communicate with instructors in order 
for the latter to answer any questions or clarify uncleamess in any aspect of the course, then learners may 
not perform as expected. In the same context, if learners do not communicate with each other to provide 
support to each other and discuss intellectual topics in synchronous and asynchronous modes, then 
instructors* expectations will not be met nor students will master the content. In addition, if both learners 
and instructors have difficulties using the hardware and software necessary for interaction, then all their 
efforts will be consumed with how to figure out how to communicate with each other and little effort will 
be spent on the actual class content. Research in this area would help educators define the set of skills 
needed to enhance interaction and satisfaction among distance education students. 

Students who lack computer skills face major frustration and may drop the course because they 
cannot deal with technology. One way to prevent students from dropping out is to organize, possibly during 
the institution’s orientation week, a one-day technology meeting, during which students are taught how to 
use an email system, the Internet, and the conferencing system used in the course. Furthermore students 
should also be taught on how to submit online assignments. During the technical orientation meeting, 
students could create and post on the Internet their personal web page that includes a photograph and a 
short biography. This is a great way for classmates and instructor to put a face to the name. The meeting 
would be also useful for students to get to know each other and the instructor of the course. 

In case students are not able to visit campus during the technology orientation meeting, the 
educational institution should provide technical support assistance via email or a toll free number. Online 
tutorials detailing the daily procedures of a course as well as describing how to use technology should also 
be developed. Institutions that created such a department and offer technical support twenty-four hours a 
day, seven days a week have lowered attrition rates that were due to technology problems. Some may argue 
that creating such a department will require a huge budget the university might not be willing to pay. 
However, funds for such a department might be available from the increased amount of students. 
Alternatively, institutions might charge students a small technology fee in order to overcome the cost 
barrier. 

Providing a teaching assistant (TA) for instructors who teach web-based courses is another 
solution to help students stay enrolled in the course and succeed. The TA could be responsible for 
providing technical support to online students. Twenty students per TA is a reasonable number of students 
for such a task. One drawback to this solution is that the TA will not be available to provide immediate 
feedback twenty-four hours a day, seven days a week. Still, some technical assistance it is better than 
nothing. 

In case none of the above is possible, course instructors should consider teaching students how to 
use technology during the first few weeks of the course. This might take away from the course content, but 
in the long run it will pay off Students are would be likely to drop the course and more likely to enjoy the 
content, and instructors will not be frustrated every time they receive a phone call or an email message 
from students who require help with technology. 

Students' prior experience with CMC has been found in the literature as one of the factors that 
influence interaction in an online environment (Vrasidas & Mclsaac, 1999). Although many students might 
be new to the online modality of delivering instruction, the more contact and hands-on experiences they 
have with technology prior to or at the beginning of the semester, the more likely students are to persist in 
the online course. 
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Research on students’ self-eflicacy beliefs with CMC 

The first family of motivational constructs is concerned with students’ perceptions about their 
ability to accomplish a task. Self-efficacy plays a significant role in predicting academic achievement in 
traditional environments. For example, Pintrich and De Groot (1990) suggested that the improvement of 
students’ self-efficacy beliefs leads to increased use of cognitive and metacognitive strategies and, thereby, 
higher academic performance. In the context of online education, increased levels of self-efficacy beliefs 
toward the technology utilized might help online students to communicate and interact with their peers and 
instructors. One way for instructors to identify students who lack the ability to use CMC in an online course 
and should provide early feedback. Students’ perceptions of their skills with using technology can be 
measured using a self-efficacy questionnaire administered by the instructor at the beginning of an online 
course (Miltiadou, in preparation). The questionnaire can be posted on the Internet and students’ answers 
could be collected either by email or using a database application. After analyzing results, instructors could 
warn students of their lack of skills with using technology and advise them accordingly. The provision of 
early feedback and remediation may result in students persisting in the course. This may translate to a 
decrease in the high attrition rates evidenced in some online courses. 

Increased levels of interaction among online students and the instructor may lead to an increase in 
students’ self-efficacy perceptions with using online technologies. Immediate and effective feedback from 
the instructor on students’ questions and assignments, as well as constant comforting of students at the 
beginning and throughout the semester with statements such as "you can use technology after a lot of 
practice’’ may raise students' comfort levels with using technology. Students' being told "you can do this" is 
one of the four principal sources of acquiring information for self-efficacy assessment (Bandura, 1986). 

Virtually pairing up students for moderating threaded asynchronous discussions may also raise 
their self-efficacy beliefs with using technology. Vicarious experience is another principal source of self- 
efficacy appraisal (Bandura, 1 986). Students uncomfortable with technology would observe their peers 
moderate online discussions and as a result, they too would learn how to do it. 

Instructional Design and Course Development Principles for Online Courses 

The abundance of web-based courses that exist today does not guarantee that these courses are all 
instructionally sound. There are a number of design and development issues and principles that course 
developers need to consider in order to design high quality courses. 

First, course developers need to answer some very basic questions such as (a) is the course content 
appropriate to be taught on the web? (Porter, 1997) (b) who is the target audience? (c) what are the course 
goals and objectives? (d) how will objectives be assessed? (e) what are the limitations of technology? 
Instructors who simply post lectures and assignments on the web are supplementing their course, but this 
would not usually constitute an instructionally sound course. Course developers should refer to 
instructional design models, procedures, and techniques and follow the necessary steps in order to ensure 
high quality courses (i.e. Dick & Carey, 1990; Gagne, Briggs, & Wager, 1992; Smith & Ragan, 1999). 
Several other good sources for advice regarding instructional design principles for distance learning include 
Eastmond and Ziegahn (1995), Hirumi and Bermudez (1996), Ritchie and Hoffman (1997), Savenye 
(1999), and Starr (1997). Additionally Hannafin and his colleagues have developed guidelines for 
designing open-ended learning environments, which might form part of the distance learning course 
(Hannafin, Hall, Land, & Hill, 1994; Hannafin & Land, 1997). 

In addition, instructional designers could follow Keller’s (1987) Attention, Relevance, 

Confidence, and Satisfaction (ARCS) model in order to develop an intrinsically interesting course which 
would enhance students’ motivation. Keller’s (1987) ARCS model would be used to develop courses that 
would capture students’ attention, enhance content relevance with their prior knowledge and experiences, 
built students’ confidence, and enhance their satisfaction with instruction and content material. 

Instructional designers should also make an effort to enhance the relevance of course contents with 
students’ educational backgrounds and experiences. Incorporating case studies that approximate real life 
situations and match students’ interests would increase students' learning goal orientation because it would 
capture students' attention and motivate them to learn. Instructors should not substitute the learning 
experience with easy assignments because students might lose interest (Locke, 1996; Locke & Latham, 
1990). As a result, students’ learning goal orientation might decrease, and it is learning goals that are 
responsible for students’ mastery of the subject being taught. 
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Interactivity should also be integrated into online courses in order for instructional designers and 
instructors to ensure success. A well designed distance education course should include many topics for 
online discussions, feedback from students, the instructor, as well as experts, and links to online sources of 
pertinent information such as journal articles, databases, and web pages with relevant to the course 
information. A strong sense of interactivity could also be emphasized with a certain degree of humor 
(Parker, 1999). Humor can make the class more enjoyable, it can generate a feeling of sincerity among the 
participants, and can lighten the burden of the learning curve for both the students and the instructor. 

Instructional designers should create well-planned and structured online courses, including syllabi 
that are clear and concise. Vrasidas and Mclsaac (1999) found that structure influences interaction in an 
online environment. Students should know exactly what is expected of them and the precise steps they need 
to follow to accomplish the objectives of the course. Instructors thus would help students control their own 
pace for finishing assignments, posting messages to various discussion questions, and reading the required 
material. Johnston (1997), for instance, recommends developing a hyperlinked syllabus, providing students 
access to many other types of resources. 

Course developers also need to either design their own interface, or use a web-course template 
provided by the distance learning department at their institution. Web page design is not a simple step and 
should not be taken lightly. The interface should be user-friendly, allowing for sufficient white space and 
consistent placement of text and images on each web page. A dark font size on white background should be 
used because often students print out web pages to be able to read the material easily. An appropriate font 
style and size, universal for both PC and Macintosh computers, is required for students to be able to read 
without difficulty. Colors should be subtle and should complement the content. Navigation should include a 
site map for easy access of all web pages by the students. 

At the end of the design and development phases, course developers should post all necessary 
information on the class web page before the beginning of the semester. Such information should include 
the weekly schedule of readings, assignments, and discussion topics. 

The issue of online exams is another major one in online courses. Often instructors feel that they 
do not have control over such examinations. One way to solve the problem is to have educational 
institutions collaborate on testing. For example, collaborating universities could provide the classroom and 
the exam proctors for students living in that area. Another idea is to have students submit essay-type 
questions, case studies, or research papers, on which it would be difficult for them to cheat. 

A final instructional design consideration is the value of conducting formative evaluation and 
revision on the distance learning course and materials, as specified in most instructional design models. 
Porter (1997) in her chapter on determining whether courses are appropriate for distance delivery includes a 
useful checklist for evaluating distance learning courses. Another example of criteria to be considered in 
evaluating distance courses in schools is presented by Hawkes (1996). 



Conclusion 

In light of high attrition rates within online distance education environments, and a growing 
number of traditional courses utilizing online technologies, traditional as well as online students are likely 
to experience distance education methods in the near future. Providing information that reveals learners* 
motivation and strategies for successful engagement in distance learning environments is likely to prove 
beneficial for both distance education and traditional education that is supported through online education 
methods. 

More research is needed in order to shed light on which motivational constructs can be identified 
as predictors of success in an online environment. The lack of sufficient studies coupled with the rapid 
growth of online courses demand more investigation in the area so that attrition rates of online learners 
decrease and learners achieve their educational goals. The rapid technological advances that are an 
everyday phenomenon at the beginning of the 21^' century offer the basis for collecting more empirical data 
in order to enable educational institutions, instructional designers, and educators to know before even a 
course begins learners' perceptions, goals, and organizational skills. This knowledge would contribute vital 
information for further quests in understanding the online modality of education and therefore help both 
lower attrition rates and increase the design of instructionally sound web-based courses. 
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Abstract 

This paper discusses a case study research experience focused on a professional medical society 
employing holistic instructional technology to solve a problem in continuing medical education 
(CME). The revelatory case study was conducted to provide an instructive/evaluative tool to 
inform future society educational initiatives, and as a possible reference for other organizations 
with similar responsibilities. The challenges emergent from the fact that the case study focuses on 
events which parties other than the researcher control are detailed, yet the experience is presented 
as essentially positive. 

Case studies, it might reasonably be claimed, are attractive to researchers who relish rich 
experience, the opportunity to collect a wealth of data from a wide and varied spectrum of sources, the 
thrill of discovering and identifying emergent themes or patterns in data, and, ultimately, answering the 
questions “how” and “why”. 

The most salient disadvantages of case-study research are widely noted: they are time-consuming 
and can produce an unwieldy body of data, validity is difficult to establish, and generalizability, at least as 
quantitatively construed, is not achievable. Nonetheless, it should be noted that every research method has 
disadvantages and that each point of criticism emergent from the identified disadvantages of the case study 
method can be countered, with a reasonable claim made that it is the most useful and valuable research 
method for certain circumstances. To the claim that case studies are time-consuming and overly data- 
productive, it can be said that time is well invested, and rich detail welcome, when exploration, description, 
and ultimately, explanation are desired. To the claim that validity is difficult to establish, it can be pointed 
out that an abundance of data serves well. In fact, the more data, the better, in regard to validity. And, 
relative to generalizability, it can be noted that well-recognized authorities on case study research, such as 
Yin (1994), assert that it is inappropriate to conceive of the term as having the same meaning as it does in 
quantitative research. 

The greatest challenge, however, for the case study researcher may arise not from the amount of 
data collected, nor from the apologetic task. Rather, it may emerge from the defining characteristic of the 
method - that the case study examines a contemporary phenomenon within its real-life context, wherein 
parties other than the researcher control events and boundaries between phenomenon and context are not 
clearly defined. (Yin, 1994) 

In the investigation that produced “The process of implementing instructional technology in 
response to a problem in continuing professional education: A case study” that was precisely the reality. 



The Case 

The Organization - CME Provider 

The case focused on events that occurred between August 1997 and October 1 998 as a sixty- five- 
year-old medical professional society dedicated to serving the research, professional and educational needs 
of its 5,000 North American member surgeons engaged in holistically-construed instructional technology to 
solve a problem in continuing medical education (CME). 
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Of all the practice-related needs that exist for surgeons, none is more compelling than the need to 
qualify for and retain board certification within their specialty. To do so, they must earn 150 CME credits 
within a three-year period, a significant portion of which must be for participation in “Category 1” 
activities, sponsored by appropriately accredited bodies. 

The society is accredited through the Accrediting Council for Continuing Medical Education 
(ACCME) as a provider of CME. The criteria for CME programs are dictated by various 
licensing/certifi cation agencies, and by the less structured, but equally demanding, exigencies presented by 
constant advances in surgical techniques and technologies. 

Organizational Challenge & Response 

Like many CME providers in recent years, the society, whose members are sub-specialty 
surgeons, has been challenged by the realities of new limits on practitioners’ time and ability to travel, to 
some extent influenced by managed care, as well as by changing demographics and practice configurations. 
These changes have eroded participation in traditional CME formats - specialty conferences, symposia or 
core-curricula courses. The result has been general concern among surgeons regarding credit acquisition. 
For a small, but growing number, specialty-board certification, and contingently, professional society 
membership has come to be at risk. 

In response, the society launched a new educational project in August 1997. This project focused 
on implementing an on-line CME program based upon readings in the society’s professional journal, and 
was enabled by the American Medical Association - Council on Medical Evaluation’s adoption earlier in 
1997 of new criteria that removed existing constraints with regard to journal-based CME. It was the hope 
of society officials, and especially the Executive Office-based Committee for On-line CME, that the 
program would contribute to solving the credit acquisition problem affecting many surgeon members as 
well as the members of three other affiliated medical professional organizations. 

Research Problem 

Though society officials and committee members had some confidence that the surgeons’ 
familiarity with the journal would be a positive factor, they also were aware that the quality, and thus, the 
appeal of the educational product would largely determine the degree to which it would be a solution to the 
problem of CME credit acquisition. Thus, from the outset, the committee members, especially the CME 
Project Coordinator and the Director of Education, were concerned with the decision- 
making/implementation process and with assuring that it would yield an impressive result. Their tacit 
understanding of the process was such that they recognized its complexity, situated as it was within the 
immediate, hierarchical environment of the medical society, and the broad environment of externally- 
imposed professional and curricular requirements, as well as other, relevant organizations with oversight 
authority. 

The society and, most specifically, the key participants in the project wished to have an 
opportunity to examine their own behavior in the decision and implementation process. Such an 
opportunity could serve both instructively and evaluatively in future endeavors. 

Purpose 

Thus, the purpose of the investigation was to provide a “revelatory” case study in CME useful 
both for the medical society in future endeavors and for other organizations with similar responsibilities 
with regard to member/client education, and, perhaps, facing similar problems fulfilling those 
responsibilities. In doing so, particular attention was to be paid to the decision-making/implementation 
process as a complex, holistic instructional technology. 

By definition, a revelatory case study involves a situation not usually available for observation and 
analysis, as in the medical society’s on-line CME project. Further, the project met the other criteria 
identified as necessary to justify the case study approach related to (a.) the type of research question posed 
(emphasizing “how” and “why”), (b.) the extent of control the researcher has over actual behavioral events 
and (c.) the degree of focus on contemporary as opposed to historical events. (Yin, 1994) 

Research Questions 

A broad set of questions was investigated in the CME project case study as is appropriate to 
qualitative work (Savenye & Robinson, 1996). These questions were designed to reveal broad themes, as 
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well as to enable discovery of specific answers. The revelation of themes allowed context to be provided 
for the specific answers in addition to useful description of the organization’s problem-solving/decision- 
making framework and educational technology paradigm. The set of research questions included: 

• How does the CME project committee view the problem-solving process? 

• What does the committee believe to be the advantages/disadvantages of the chosen medium 
for instructional delivery? 

• How does the committee view desired outcomes and approach assessment of learning? 

• How congruent with a holistic definition of instructional/educational technology is the 
society’s problem-solving process? 

Review of Literature 

The review of literature for the case study examined scholarship in a range of knowledge areas in 
keeping with Marshall and Rossman’s (1989) observation that qualitative studies usually have several 
strands of related literature. The literature presented was grouped into three categories: CME topics; 
organizational decision-making, problem solving, and policy topics; and, program-related topics. 

Methodology 

The application of method involved relating to the subjects (the committee and other parties) and 
the phenomenon (the CME project) as a passive to moderately-participating observer (Spradley, 1980). 

The level of participation engaged in at any particular time was selected to allow achievement of the 
significant purpose of qualitative research: “to discover important questions, processes and relationships, 
not to test them” (Marshall and Rossman, 1989, p. 43). The investigative techniques employed, in support 
of and in addition to participant observation, were: recording fieldnotes, audiotaping meetings, conducting 
structured and unstructured interviews (recorded by note and audiotape) and collecting documents. 

The data analysis rested upon a foundation of patterns revealed in the text of fieldnotes, 
audiotape transcripts and organizational documents. Through the following series of steps, the assembled 
data were transformed into meaningful interpretation, framed by the need to respond to the research 
questions: 



• Reduction of the body of information/selection of significant points. 

• Identification of regularities and patterns, 

• Development of coding categories through which to represent regularities and patterns. 

• Determination of a method of “display”. 

• Drawing conclusions. 

• Verifying conclusions. 

The review, reduction and selection of data and the identification of regularities and patterns 
allowed the development of a list of coding categories, the nomenclature and relationships of which were 
structured in accordance with a definition of educational technology developed by the Association for 
Educational Communication and Technology in 1977 (cited in Gentry, 1995). This definition was selected 
as it is multi-factoral and cognizant of the complex social aspects of educational technology as a holistic 
process: 



Educational technology is a complex, integrated process involving people,procedures, 
ideas, devices and organization for analyzing problems and devising,implementing, 
evaluating and managing solutions to those problems involved in all aspects of human 
learning, (p. 4) 

This approach to developing coding categories was chosen when it appeared upon review of the 
data that the emergent themes and patterns were closely congruent with the terms of the definition. 
Admittedly, when the case study was proposed, it was anticipated that the presumably complex project 
decision-making and implementation process would be reflective of an holistic conception of educational 
technology, and, in fact, evaluated as such; however, it was only upon examination of the data that the 
degree of congruence became evident. Thus, the discussion of data was presented under three thematic 
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headings: the Social-Professional Context theme (which included data coded as “people”, “procedures”, 
“ideas”, “devices” and “organization”), the Problem Analysis theme, and the Problem Solution theme 
(comprising data coded as “devising”, “implementing”, “evaluating” and “managing solutions”. 

Responses, Conclusions, and Recommendations 

The research data - notes from observed and participatory events; audiotapes and transcripts from 
meeting and interviews; and documents, generated both within the society and by outside entities - 
provided the main body of text in the response section. In addition, though, points from readings featured 
in the Review of Literature were brought to the question-response task where appropriate for purposes of 
comparison/contrast with, and evaluation of, the decisions and practices of the project. 

Conclusions drawn from the analysis of data reflected the view that the society engaged 
competently in an holistic technology resulting in the on-schedule implementation of an on-line CME 
program to meet the credit acquisition needs of an increasing number of surgeons. Nonetheless, the data 
also supported concluding that various organizational and practical realities compromised achieving an 
optimal outcome. Among those realities were: 

• the lack of consideration of options to on-line journal-based CME; 

• the influence of broad environment politics and competitive relationships on decision- 
making; 

• the organization’s committee structure which both enhanced and impeded decision- 
making and problem-solving efficiency; 

• budgetary constraints which resulted in foregoing an originally-planned interactive 
feature of the on-line program in the initial offering; 

• and, organizational uncertainties related to surgeons’ attitudes toward on-line CME, 
which limited the allocation of resources in the implementation and marketing of the 
program. 

It was further concluded that the CME project, even with its imperfections, can serve in a useful, 
revelatory and instructive manner for the society and other professional organizations faced with similar 
tasks related to providing professional education. 

Recommendations were provided primarily as a guide for future society CME initiatives. The 
most significant were that the organization: 

• enhance environmental scanning relevant to CME and consider all available options; 

• improve communication within the various organizational offices and committees; 

• improve the detail and timeliness of financial/budgetary information; 

• and, prioritize the development of CME programs delivered through nontraditional 
formats. 

Research Challenges 

Knowing that case study research is a complex enterprise presenting the commonly articulated 
challenges related to time expenditure, the amount and nature of data and concern for establishing validity 
and meeting questions on generalizability, probably will not prepare a researcher for the particular 
challenges of a specific investigation. As indicated earlier, the particular challenges of “The Process of 
Implementing Instructional Technology in Response to a Problem in Continuing Professional Education” 
related to one of the defining characteristics of the method - that the case study examines a contemporary 
phenomenon within its rcal-lifc context, wherein parties other than the researcher control events and 
boundaries between phenomenon and context are not clearly defined. The challenges due to lack of 
researcher control in the CME project case study can be categorized as: delays, scheduling difficulties, 
researcher role issues. 
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Delays 



The journal-based CME program, albeit a compromised version, was announced at the society’s 
board meeting in October, 1998 and the first credit-designated reading and test appeared in the society’s 
January, 1999 journal. Though both events occurred on schedule, the period of decision-making and 
implementation had been marked by delays from at least two sources - external parties and physician 
committee members. With regard to the former, the most noteworthy delay was related to clarification on 
what constitutes credit-worthy journal-based CME. For several months during the decision phase of the 
project, key parties at the medical society sought direction, in vain, from the ACCME and its then-new 
managing body, the American Medical Association (AMA). Speculation within the specialty society was 
that “politics” and competition (the AMA is a CME-provider, as well) were factors. 

Other delays stemmed from the fact that committee members based at the society’s Executive 
Offices were not the only decision-makers or involved parties in the CME project. In addition, member 
surgeons served on the project committee and on other relevant committees (such as the CME core- 
curriculum committee) and boards (including the journal editorial board and the finance board), which 
sometimes had oversight responsibilities regarding certain aspects of the on-line CME initiative. The input 
of these surgeons was needed for many decisions; for example, those related to; the range and sequence of 
medical specialty topics for CME-designated journal articles, article authorship, examination question 
writing, examination rigor, quality assurance, and budgetary matters. Due to the demands of practice, key 
surgeons were not always able to respond promptly to inquiries and requests from executive office 
committee members for consideration of various points and issues. This impacted virtually all forms of 
data - event observations, interviews and documents. 

The effect of these delays was to stall certain aspects of the research and writing processes. 
Though, as previously acknowledged, the on-line CME project was completed on time (as was the 
research), the timing of data-gathering generally was not controllable. Thus, it was not always possible to 
adhere to a work plan, or nor make the most efficient use of time. Sometimes, there was relatively little to 
do, while at other times there was an inundation of data that had to be rather quickly reviewed, selected, 
categorized and analyzed. 

Scheduling 

Another set of research challenges emerged from certain unpredictabilities in the schedules of 
Executive Office-based committee members. Though all key parties were cooperative with the research, 
allowing observation of all relevant meetings and participating in lengthy interviews, the scheduling of 
such events often changed, sometimes suddenly. Though some of these changes were linked to the 
previously discussed delays imposed by outside parties, many were not. Rather, key committee members, 
each of whom had a variety of responsibilities beyond the on-line CME project, were subject to changes in 
travel plans, the rescheduling of other appointments and meetings, and occasional minor, but nonetheless 
demanding, “crises”. In addition, during the time that work on the project was underway, the Project 
Coordinator’s relationship to the society (though not the project) changed, resulting in less in-office 
presence. 

All of these realities required considerable flexibility and willingness to operate under less-than- 
ideal conditions on the part of all parties. In addition to the obvious difficulties- those posed by the need to 
reschedule contacts - other, more idiosyncratic ones arose. For example, one lengthy interview was 
conducted in a restaurant, to accommodate the travel and meeting schedule of the society’s Director of 
Education. Fortunately, the audiotape, complete with background conversations and the clattering of 
dishes and tableware, was decipherable. Another interview was conducted in the home of the Project 
Coordinator who had become a “Special Projects Consultant”, an adjustment made to better meet the 
demands of parenting. The audiotape collected from the two and one-half hour event is replete with sudden 
stops, starts, wails and assorted loud background noises. Again, though, a useful transcript with 
surprisingly coherent text was produced. 

It should be acknowledged, however, that free access was allowed to the society’s Executive 
Office resource center and archives, so research time suddenly made available due to rescheduling 
sometimes could be used for reviewing educational and other materials for context. 
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Researcher Role Issues 

Challenges accompanied the researcher role in the CME Project case study, as well. The first 
relates to the necessity of being ever mindful of the boundaries of the researcher/subject relationship and 
striving to maintain the appropriate degree of participation in events. While this was not a significant 
problem during data collection, there were times, for example, when discussions or decisions focused on 
some area of personal knowledge or experience, when the temptation existed to contribute more than what 
might have been appropriate given the level of participation chosen. 

In addition, because it is within the range of researcher responsibilities to resolve or find 
explanations for seeming contradictions in data, it is an ongoing test of attentiveness to detail, persistence 
and tact to stay abreast of and clarify such occurrences. In the CME project case study, contradictions 
related to such important subjects as: the likelihood of practitioner acceptance of non-traditional formats, 
the level of financial commitment to the project, the interactive feature of the on-line delivery system, the 
selection of writers for examination questions, program competition and marketing. Some contradictions 
were self-contradictions by key project participants, while others were contradictions between participants. 
Achieving clarification involved a variety of methods, including keeping a list of points to include in later 
interviews or discussions, checking the text of newly-released documents, or simply making a call to a 
particular, informed individual. 

Finally, writing the conclusions and recommendations may have presented the most significant 
“researcher role” challenge in the CME Project case study. It cannot be said that any important point was 
omitted, nor was there any conscious lack of forthrightness. However, because a prior consulting 
relationship had existed (albeit occasional rather than continuous), because the project committee members 
had been exceptionally cooperative, given their professional responsibilities both in regard and in addition 
to the project, and because the key parties would read the document, selecting the appropriate terms and 
tone of expression may have been a more tedious process than would normally be the case. 

Conclusion: “Out of Control” As a Positive Experience 

The fact that much is out of the researcher’s control in conducting a case'study is, after all, 
known at the outset; it is of the essence of case study research. Yet to an extent not offered by other 
methods, conducting a case study offers the researcher the opportunity to have an intimate view of what 
real human beings do in real situations, to observe ambiguity, to understand the multiple influences brought 
to bear on decisions and implementations, to appreciate complexities, and ultimately, to witness how things 
happen or are accomplished, for better or worse. 

The researcher planning a case study might be well advised to prepare for the volume and 
richness of the data, though also, for the sometimes frustrating, but often surprising and enlightening ways 
in which it emerges, for the unpredictability of its content, but, most of all, for the fascination and challenge 
of the “out of control” aspects of the method. 
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Abstract 

In an era of restructuring many organizations are downsizing and merging. There is an explosion 
of information, and workers are required to cross-train and re-educate themselves. This re- 
education must be both time efficient and cost effective to meet the needs of the organization and 
the worker. Nursing is one of many professions that is faced with this dilemma. Strategies for 
delivering continuing education which is realistic, non-threatening, and easily accessed by the 
professional in the work place are essential. One possible solution for this problem is computer 
based training on the unit. This paper will describe the development and implementation of a 
case-based program designed for a workstation, A brief description of the preliminary results of a 
qualitative research to determine the efficacy of compute-based training in the workstation will 
also be presented. 



There have been many changes in organizational environments over the past several years, 
including downsizing and mergers. There are many factors that contribute to these changes and the 
changes effect the whole organization (Volpe, 1999). Another change that has occurred in recent years is 
the explosion of information which changes the knowledge base that the worker must have in order to 
perform adequately in the new environment. Workers require training or further education to be prepared 
for new, evolving knowledge. This re-education must be time efficient and cost effective to meet the needs 
of both the worker and the organization. 

Several obstacles must be overcome for this re-education to be effective. Many organizations 
have limited classroom space available. There is difficulty getting workers released from the work area to 
participate in education and/or review sessions. Poor turnout rates further add to this dilemma. Make-up 
classes for workers who missed important sessions require additional time and money. 

One organization that is effected by restructuring is the hospital, and one group of workers that is 
affected by this is nursing. There are fewer registered nurses (RN’s) to plan and direct the care of the 
patient while they are responsible for cost containment (Dienemann & Gessner, 1992). Many nurses are 
working part-time as well as working non-traditional shifts, such as ten or twelve hour shifts. Nurses are 
also required to be cross-trained in many areas of the hospital rather than working in one specialty alone. 

The need for new modes of continuing education delivery that are realistic and non-threatening is 
a problem that must be solved. Diverse learning strategies or technologies are required to accomplish 
appropriate continuing education. One of the technologies that have been shown effective in several 
studies in both industry and the military is computer-based instruction (CBI) (Griddle, 1995; Holloway, 

1 996; Kearsley & Lynch, 1 994). This paper will look at the development and implementation of a 
computer program on the patient care of a client with a pulmonary artery pressure catheter for use in the 
workplace by the RN for continuing education. A brief description of the preliminary results of a case 
study to ascertain the effectiveness of this method of education at one community hospital will be 
presented. 

Computer-based Instruction 

CBI can deliver instruction via several methods, including: drill and practice, tutorials, and 
simulations. Drill and practice was one of the first methods used for CBI in the 1970s. “These electronic 
ditto sheets presented problems — most commonly mathematical problems — for learners to solve” 
(Jonassen, 1996 p. 4). A second form of CBI is tutorial. “The archetypal tutorial would present some 
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information in text or graphics and then ask the learners a question to assess their comprehension of what 
had been displayed (Jonassen, 1 996 p. 5). This information can be used to deliver new information, or to 
reinforce learning from previously-learned information. While this does not teach problem solving it is 
more advanced than the drill and practice method of CBI. The final form of CBI is the simulation. 
Simulations are case studies that allow the learner to attempt to solve a problem in a non-threatening 
manner (Riley, 1 998). Simulations can help with the transition between theory and practice in nursing. 
Simulations are an effective method of teaching basic nursing, as well as for teaching CE. 

Advantages and Disadvantages of CBI 

CBI does not replace instructors; it is a tool that can be used to facilitate learning. CBI can also be 
used for evaluation purposes. This section will look at advantages and disadvantages of CBI in nursing 
education. 



There are many advantages to CBI. Simulations allow for the compression of time, and students 
can see the results of their actions using CBI (White, 1979). Results can’t be seen this way when working 
with actual patients. CBI is also a safe learning environment to practice problem-solving skills (Howard, 

1 987). A safe learning environment consists of the student feeling safe to make mistakes, and the client 
being safe from these mistakes. CBI allows all learners to experience the same situations, to receive 
immediate feedback, and it is individualized to suit the learner (Howard, 1987). 

CBI is also cost effective for continuing education (Criddle, 1995; Pogue & Pogue, 1998). 
Evaluation of students, as well as record keeping, is another advantage of CBI (Ball & Hannah, 1984). 
Further more students have decreased anxiety with CBI because of the level of privacy. 

Disadvantages of CBI 

Disadvantages also exist with CBI. One of the disadvantages is the lack of appropriate computer 
software, especially for CE, and the cost and time required to author CBI (Ball & Hannah, 1984; Pogue & 
Pogue, 1998). Another major disadvantage for CE is the lack of computer competency by many staff 
members (McDaniel, Matilin, Elmer, Paul, & Monastiere, 1998). The lack of home computers may affect 
programs that are to be taught at home. Software should be evaluated before purchasing to determine if it 
is appropriate for the learning objectives to be accomplished. Instructors must be trained to use the 
program or computers before the staff can use it. To detect any problems, observation of the staff while 
using the program is helpful. Finding practical way of integrating the program into the workday of the staff 
is one additional difficulty that must be addressed. 

Constructivist theory using case-based format 

For Duffy and Cunningham, constructivism is a term that refers to a variety of views including 
“(1) learning is an active process of constructing rather than acquiring knowledge, and (2) instruction is a 
process of supporting that construction rather than communicating knowledge” (Duffy, 1996 p. 176). The 
constructivist theory allows the students to build from their own knowledge base and discover the correct 
answers on their own rather than being told what is important. Learning is an active process that allows the 
students to direct their own learning. They discuss problem-based learning (PBL) and case-based learning 
(CBL) as instructional models using problems as a focus. 

Case-based learning is a method of instruction that presents cases to the students in order to 
illustrate the purpose or the problem that the students will work with. There are several steps necessary to 
make case-based learning effective. The first step in CBL is to identify the problem. The problem can be 
presented in many ways, including: written (Wassermann, 1 994), video (Robison, 1 992), multimedia 
(Draude, 1996), computer (Jonassen, 1996), actual patients (Irby, 1994) or literature (Griffith & 
Laframboise, 1998). The cases should be real world situations that would be encountered by a practitioner. 
There should be a conflict or unresolved dilemma presented in the case. The students must identify the 
problem or conflict. The second step is to analyze the problem in order to determine what action to take. 

At this time the students decide what should be done next. They are guided by questions provided by a 
facilitator. This is a more passive role than the traditional instructor as expert. The final step of the case 
study is a debriefing of the case, which is usually accomplished with the entire class (Wassermann, 1992). 
This allows the students to reflect on what influenced their decisions. 



Advantages 



234 



BEST COPY AVAILABLE 




Development of computer program for continuing education 

Combining CBI and CBL to develop a program for continuing education was the first step in the 
case study to determine the effectiveness of this method of education at one hospital. A computer program 
using CBL as the theoretical base was developed by the researcher and checked for validity by a critical 
care clinical specialist and an experienced critical care nurse. Four doctoral students, in Instructional 
Technology at Northern Illinois University, developed the program. The developers included a nurse, a 
graphic artist, an English major, and an educator. The subject for the computer program is the pulmonary 
artery pressure catheter. This subject was identified as an area of training needed by the nurses in the 
intermediate care area, allowing them to work in the intensive care area as well. 

Authorware 3.5 was used to develop the program because of the versatility of the software to 
animate, include video, and allow tracking. The program uses a constructivist format. After initial entry 
into the program the nurses must sign in. This allows the researcher to track the progress of each nurse 
during his or her time in the computer program. The nurse may then receive instructions on the program or 
advance to the main screen. This screen uses a scene of a nurse’s station which allows the nurse to 
navigate in the program in a constructivist manner. The nurse can begin by caring for a patient or begin by 
going to the filing cabinet and exploring the tutorial, resources or the dictionary. The nurse may also 
decide to go to the phone and talk to an expert. The final option that can be accessed from the main screen 
is the notes area. The notepad is accessible from every screen and this allows the nurse to take notes on 
information that they would like to print to obtain a hard copy of the information. 



Several videos are utilized in the program. Three videos are experts describing the topics on the 
main screen. Other videos are found in the tutorial section, and include videos of the procedures necessary 
to care for a patient with a pulmonary artery pressure catheter. Several of these videos are taken in a 
simulated situation. The final video is of the actual cardiac output procedure performed by a nurse. The 
videos add authenticity to the computer program. They also allow for a different delivery format. 

The tutorial has four sections, including: hemodynamic physiology, description of the catheter, 
catheter insertion and readings, and hemodynamic measurements. The physiology section uses graphics 
and audio to describe the normal physiology of the heart with emphasis on hemodynamics. The second 
section has a graphic of the catheter with sections that describe each part of the catheter. Nurses may 
choose different parts of the catheter and learn more about the function of each part. The third section 
shows the insertion of the catheter, describes the nurse’s role during the insertion, and shows normal 
readings and waveforms. The final section of the tutorial demonstrates how to take measurements with the 
catheter. This illustrates the care of the patient with the catheter in place. This section utilizes video, 
graphics, and text to illustrate the care of the patient with a pulmonary artery pressure catheter. 
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Figure 2. 
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Pulmonary Catheter in the Pulmonary Artery 




Other features of the program the nurse can utilize are the dictionary, resources and the notepad. 
The dictionary defines words used in the program. A written definition is given to the words as well as 
pronunciation for the more difficult words. The resources page list web sites as well as books for 
additional resources for discovery learning. The web sites are not active in this version, because the 
hospital does not allow web access to the staff nurses. Nurses can access case studies through simulations. 

The simulations consist of three case studies of typical patients who may require a pulmonary 
artery pressure catheter during their treatment. The cases include the three most common reasons for 
requiring this hemodynamic monitoring including: a myocardial infarction, congestive heart failure, and 
septic shock. The three case studies have situations which present problems that the nurse must discover 
and treat for the patient to progress. The problems include situations with problems during insertion, 
problems during routine care, and complications that may occur during the course of care. The nurse must 
respond to the problem, or they will be given guidance by the program. The nurse may return to the 
tutorial for further information at any time during the program. 

For purposes of research the program tracks the nurses progress from the time the sign in until 
they leave the program. This allows the researcher to determine the time spent on each screen, the order of 
the progress, and decisions they made during the program. The researcher can determine the types of 
decisions the nurse makes, as well as their critical thinking in regard to three patients’ care. 



Research 

The research was a qualitative study using a case study approach at one community hospital that is 
in the midst of many of the changes described for organizations today. The research took place over a six- 
month period from August to December, The research was undertaken at the request of the clinical 
specialist, who wanted to ascertain if CBI would be an appropriate method of continuing education for their 
institution. The program was written according to the policies and procedures of the hospital. 

The nurses were asked by the clinical specialist to complete the computer program and to 
participate in the study. The hospital wanted pre’ and post-tests as part of the research. The pre-tests were 
completed before the program was brought to the hospital. The first problem occurred when information 
systems said that they would not allow the program to be installed on hospital computers. The researcher 
then brought two computers to the hospital. One was in the break room, and the second computer was on a 
cart, which could be moved from the nurses’ station to a quiet room just down the hall from the nurses’ 
station. The nurses were to work on the program during work hours. 

There were thirty-five participants in the study, including: twenty-eight nurses, four 
administrators, four information systems employees and one physician. These participants were 
interviewed after they completed the program. Three nurses from the study were observed caring with 
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patients with pulmonary artery pressure catheters, in addition to the interview after they completed the 
program. 

Of the twenty-eight nurses, fifteen completed the study. Of these nurses, six were full time and 
nine were part-time. Two nurses dropped from the study, one due to the birth of her first baby, and the 
other nurse was in an automobile accident and broke her wrist. Nine nurses elected not to complete the 
study, but were willing to be interviewed. Of these nurses one was part-time and eight were full-time. 
Computer literacy of the nurses who completed the study ran a full range from very little to computer 
literate. A scale of one to five was used for computer literacy. Nurses who knew only what was necessary 
to do what they had to at work, were given a level of one. Nurses who knew word processing were given a 
level two. Level three nurses knew how to e-mail as well as the previous skill mentioned. To attain a level 
four, the nurses also knew how to use a spreadsheet. Nurses with a level five could perform the previous 
skills, as well as work with databases. Of the nurses who did not complete the study, only one was 
computer literate. Three of these nurses did try the program; the other six did not try the program. Two 
additional nurses said they finished the program; however, tracking showed that these nurses only did the 
tutorial. Computer literacy was not dependent on the age of the nurse, as was previously assumed by the 
researcher and the administrators. 

Some of the preliminary findings obtained in the interviews follow. Nurses who knew they would 
be working in ICU were more highly motivated than those who did not perceive this as a need in their 
present workplace. Many of the nurses’, who tried the program, even though they were not computer 
literate, did like using the computer as a way of learning. Most of the nurses would have liked hands on 
learning to follow the computer program. This, unfortunately, only happened with a few of the nurses. 
Many nurses did not like working at the nurses* station because of the distractions. All of the nurses would 
like a place on or near the unit for working on the computer program. One of the biggest issues mentioned 
by the nurses was the lack of available time to complete the program. They would have liked to complete 
the program all at once, rather than in segments. 

Figure 3. 



Participants who completed the project 




Participants who did not complete the 
project 




Preliminary findings from the administrators indicate they feel CBI would be a cost effective and 
an ideal way to promote continuing education. They feel that nurses need to improve their skills on 
computers, and these skills will come as a result of the new patient care system starting in the next few 
months. They believe that people who are uncomfortable with computers will leave once patient charting 
must be accomplished on the computer. The administrators would like to see computer programs available 
on the Intranet throughout the hospital. 

The last group to be interviewed was information systems. Most of the people from information 
systems work for the corporation, rather than for the hospital. They feel that everything should be 
standardized among all of the hospitals in the corporation. All computers will be standardized and there 
will not be any CD-ROM drives on the computers. All programs that on loaded on the network will be 
done by information systems. The corporation must check them out before they are put on any system. 

This will be done at the corporate level and not by the local hospital. Nursing continuing education is not a 
priority for information systems at this time, because they are trying to get the patient care systems started 
in all hospitals first. 
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Nurses scores from the pre-test to the post-test showed great improvement. Tracking showed that 
most nurses did the tutorial before they did the case studies. Nurses who tried the program and did not 
finish usually did the tutorial first. Most of the nurses did not use the notepad. The computer that it was to 
be on was connected to a printer; however, when the researcher’s computers were brought in, there was no 
access to printers. 

Further analysis of the information gathered during the six-month period will continue for several 
more months. The researcher will review each interview and code the information from the interviews, 
observations and journals. Codes or trends will be identified developed into the main themes. 
Triangulation will also be used as surveys and other pertinent data is examined. The participants will 
verify this data as a final phase before the results are reported. 
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Abstract 

Relationships between the confidence students expressed toward using core computer 
applications: organismic factors of gender and ethnicity; academic factors of reading and 
mathematics ability and goal for attending college; and the post enrollment factor of planned 
employment during college and the students* subsequent success in community college as 
measured in terms of persistence to the subsequent semester are reported for 1434 entering, 
traditional and nontraditional community college students. 



Introduction 

With slightly over one third of recent high school graduates and additional nontraditional students 
electing to enroll in two-year colleges, higher education researchers and administrators face the ongoing 
challenge of addressing large initial enrollments in community colleges that are frequently not sustained the 
following semester (National Center for Education Statistics, 1 998). Despite study and retention strategies, 
such as sorting to identify students who may exhibit high risk pre-enrollment factors early in their college 
careers to deliver post enrollment strategies of supporting, connecting and transforming (Beatty-Guentner, 
1994), maximizing retention continues to be a critical research concern. 

A recent study on the relationship of confident computer usage to retaining traditional, entering 
community college students (Sherry and Sherry, 1997), indicated two indicators significantly related to the 
students' second semester persistence: 1) reported ability to use spreadsheets for college assignments and 2) 
high or low high school GPA's and reported ability, subsequently, in using two or three computer 
applications. 

Purpose 

Given the increased emphasis on computers at all levels of education, coupled with the need for 
studies about the role of computers in education and the direction indicated by the Sherry and Sherry 
(1997) results, this study sought to determine the relationship of pre-enrollment computing factors of 
entering community college students to their subsequent success in college. Specifically, their confidence 
in using core computer applications, as well as their current and planned use of computers and their access 
to computers were measured in relation to their enrollment the subsequent semester. Traditional 
prccnrollmcnt issues of age, gender, ethnicity, and college aspirations were also investigated along with 
academic factors of entering levels of reading and mathematics achievement and college major, and post 
enrollment factors of employment during college. 
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Method 



Participants 

Participants consisted of all students tested for initial fall semester placement at a large, multi 
ethnic community college located in the western part of the United States. Of the initial 1 ,600 participants, 
1434 provided complete and relevant data sets. Traditional (under age 20) and nontraditional (ages 20 and 
older) students were included to best represent the typical population of a community college and to 
illuminate any possible differences that might appear between the traditional students who had the greatest 
likelihood of ongoing exposure to computing experiences in school (Becker 1991; Quality Education Data, 
1994) and the nontraditional ones. 

Instruments 

CAPP Survey. Demographic data was gathered on the Computerized Assessment and Placement 
Programs (CAPP) Survey during the students’ participation in the regular orientation, testing, advising, and 
registration sessions conducted from May through August at the college. The CAPP form provided the 
means for collecting data on organismic factors of age, gender and ethnicity; the academic factors of high 
school GPA and number of academic credits and major planned; and attitudinal factors of desired 
educational goal and importance of education to self and others. Using the additional question option on 
this form, 10 computer-related questions were also posed. The questions related to levels of computer 
confidence for using word processing, databases, spreadsheets, graphics programs, and online usage 
evidenced a reliability coefficient of .98 during an earlier pilot study that employed a test-retest approach. 
The final, additional questions were designed to attain descriptive data on current and planned computer 
usage and access patterns. Response choices were limited to offer a view of a confident or nonconfident 
user. The influence of prior use of computers has shown significance in relation to positive attitude toward 
computers, whether defined in terms of computer skills (Busch, 1995; Woodrow, 1992) or computer 
accessibility (Loyd and Gressard, 1984). 

Reading and Mathematics Tests. 

Entering reading ability was assessed using the Nelson Denny ReadingTest, and for mathematics, 
with Mathematics Tests I and II developed within the Mathematics and Science Division of the college. 

Measuring Success in College 

Success was measured by a commonly used mean persistence from one semester to the next 
(Bonham & Luckie, 1993; Fralick, 1993; Halpin, 1990). Verification of the students’ enrollment status was 
obtained at the beginning of the spring semester. 

Data Analyses 

Data was analyzed using the SPSS for UNIX statistical package online. Frequency distributions 
were run to check initial data input and coding and to offer an overview of participants by types of 
responses. To investigate whether the dichotomous dependent variable, persistence in college, and predictor 
variables under study are independent. Chi-square analyses were performed. Potential interactions between 
continuous data from the total computer confidence score and college GPA and from high school GPA 
were examined using one-way analyses of variance. To examine the relative importance of the pre- 
enrollment factors in relation to the dichotomous dependent variable of persistence in college, logistic 
regression analysis was used in the forward stepwise mode. 

Results and Discussion 

Frequency distributions show a strong pattern of persistence to the following semester (74.5%). In 
regard to the organismic factors under investigation, the typical student is female (56.6%), 17 to 19 years 
old (61 .6%) of Filipino (25.6%) or Other ethnicity (27.5%). Although this pattern is typical in the 
geographical area of Hawaii served by the community college where the study was conducted, increasing 
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numbers of Hispanics, Blacks, Indians and people of mixed ethnicity indicate that the "Other" ethnic 
category reflects the trend toward increasing ethnic dispersion in the state. 

In regard to entering academic factors and the post enrollment factor, the typical student, who 
elected the to participate in the Nelson Denny Reading, scored in the college level reading range on that test 
(57.2%) ensuring eligibility for enrolling in English 100 level courses. For those students selecting the 
mathematics test options, the typical student scored at the second highest cutoff point on the Mathematics I 
Test (37.3%), thus placing in developmental mathematics courses and at the third highest cutoff point for 
the Mathematics II Test (45.0%). The typical respondent also reported an earned high school GPA of 3.0 to 
4.0 (73.3%); and plans to transfer to a four-year college (73,3%) and to work 1-20 hours per week (43.7%), 

Computer Factors 

Viewing the typical respondent in terms of computer-related factors provides a picture of a 
confident word processing user (75.5%). Email communications (42.2%); online searching (41.5%); 
spreadsheet (37.7%), database (36.9%), and graphics usage (35,3%) are not apt to be applications the 
typical respondent reports being confident using. When these applications are considered collectively, 
however, confidence with three or more of these computer applications appears (51,0%), Access to a 
computer does not appear to be a concern (73%). A plan to use a computer as one way to carry out college 
assignments also appears (82.1%) with access being at home (57.3%), 

A picture of a relatively knowledgeable computer user, thus, emerges from the self-reported data 
that is supported by the acknowledgement of current use of a computer for school or work (53.8%), In a 
similar study conducted by the authors in 1993, close to 30% reported being unable to use at least one 
computer application. Given the growth of computers in our society, this increase is not surprising. 

Results of Univariate Analysis 

An examination of each factor independently of the others--a univariate analysis-using the Chi- 
square statistic, indicates that age, ethnicity. Nelson Denny Reading score. Mathematics I score, high 
school GPA, anticipated college major, goal for college, database usage, and planned employment during 
college are significantly related to persistence in college. 

Among the ten organismic, academic and goal-directed, and post enrollment factors studied, all 
but gender and the Mathematics II score are significant. Students in the age range for recent high school 
graduates (17 to 19 years old) persisted at a higher rate than those students twenty years old or older, 

Asians persisted at the highest rate. 

Students who scored at or above the cut off score for entrance into college level reading courses 
and college level mathematics courses on the Mathematics I entrance test also have higher persistence rates 
than their peers who earned lower scores, A similar pattern emerged for the small number of students who 
took the Mathematics II Test, but these results did not reach levels of significance. Surprisingly, students 
reporting low high school GPA's are more apt to continue to the subsequent semester than students 
reporting average or high GPA's are. 

Students who noted their intent to enroll in a major other than Liberal Arts are somewhat more 
likely to persist than those who indicated that Liberal Arts was their planned major. Although significant, 
this factor of college major needs to be interpreted cautiously. Indicating a major during the orientation and 
testing process does not necessarily mean subsequent selection of that major. 

Having a goal of transferring to a four-year college relates highly with persistence. Plans for being 
employed while enrolled in college, while related to retention, show little differentiation between working 
or not until work time exceeds 40 hours. 

Among the twelve computer factors, only two, confidence using databases and current use of 
computers for school or work, are significantly related to retention in a positive way. 

Results of Analysis of Variance 

Given that the earlier Sherry and Sherry study (1997) found a significant interaction between 
persisters’ and nonpersisters’ high school GPA scores and their computer confidence levels, an analysis of 
variance (ANOVA) procedure was carried out with the current data to examine this interaction. Results 
show that the mean for each of the three types of GPA scores (high, average, and low) are not significantly 
different for persisters and nonpersisters in relation to their computer confidence levels. 
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Results of the Logistic Regression 

To investigate the multivariate effect of all factors together that were identified during the 
exploratory phase of the data analysis process as being potential predictors of persistence in college, 
logistic regression was employed using a forward stepwise approach 

The results of the analysis show that of the ten factors entered into the equation, four remained. In 
order of their importance to persistence in college, they are educational goal for college, Mathematics I 
score, ethnicity, and current use of a computer for school or work. Overall, the resulting model correctly 
classified 73.7% of the students. 



Summary 

Computer Factors 

The results of the analyses of the relationship of the computer-related factors, including the 
interaction of students’ overall scores of confidence using multiple computer applications in conjunction 
with their high school grade point averages to persistence in college show that only one factor— current use 
of computers for school or work— is significant in both the separate and multivariate analyses. 

During the univariate analysis, confidence using one specific computer application, databases, was 
significant. The earlier Sherry and Sherry study (1997) had a similar finding. In that study, significance was 
found for only for one application, use of spreadsheets. Although that factor was not significant in this 
study, it may be that the influence of either of these two programs may be a reflection of the logical 
thinking needed to work with them. 

The respondents’ current degree of involvement with one of the two programs at the time of a 
study may affect the outcome of which one attains a degree of significance, rather than databases or 
spreadsheets per se in relation to retention in college. 

It is possible that respondents may have underestimated their confidence when asked about the 
separate core computer applications. The positive relationship between their reported actual use of 
computers for school or work to success in college suggests the possibility that transfer of this computing 
ability to their subsequent academic work during the semester under study may have occurred. 

Other Factors 

Ethnicity is related to the persistence pattern for the traditional entering students studied earlier at 
this same community college. In both studies, Asians had the highest persistence rates. Compared to the 
Asian students, Hawaiians/part Hawaiians/Pacific Islanders and students classified as Other Ethnicity, 
Whites and Filipinos are associated with increased log odds of persistence in this study, although the effect 
for the latter two ethnic groups did not reach significance. In the earlier study, students with these ethnic 
backgrounds were more likely to become nonpersisters. The current findings reflect the mixed results in 
retention studies. The positive persistence of the Asians, Hawaiian/part Hawaiian/Pacific Islands, and 
Others reflect the higher retention rates found among ethnic groups that predominate in the geographic area 
served by the college. (Feldman, 1993). When this study was conducted, however, Filipinos comprised the 
largest ethnic group. Similar to what Webb (1989) found when he studied retention in Los Angeles 
community colleges where African Americans were the majority, the Filipinos in this study, in comparison 
to the Asians, did not reflect significant rates of retention. Pascarella’s and Terenzini’s (1991) meta 
analysis of factors related college retention found that minorities do have lower retention rates if they 
begin at community college. Although they represented the largest ethnic group in 1996 at the college, 
when the Asian students are combined with those students identified as Other (many of whom are of mixed 
ethnicity), their combined presence may make the Filipinos feel as though they are a minority. 

Similarly, educational goal is related to the findings from the earlier Sherry and Sherry study 
(1997). Transferring to a four-year college was highest on the selections made by both student groups. In 
comparison to that goal, however, personal enrichment, other, a vocational/technical degree, and a two-year 
degree were important, respectively, to retention in this current study. The latter two did not reach 
significance. Only the “Other” factor appeared positively in the previous study. All the other factors were 
related to nonpersistence. The decline in the economic situation in Hawaii since that earlier study may 
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account for these current students’ positive connection between their educational goals and their success in 
college when they make the decision to spend time and money in college. 

The investigation of the role played by mathematics ability was supported. A positive, significant 
relationship was evident between those students whose scores on the Mathematics I Test qualified them for 
college level mathematics and their continuance the following semester. When compared to this group, 
students who achieved scores placing them in levels of developmental mathematics-Mathematics 25 and 
Mathematics 24-and remedial mathematics are retained at significant levels, but with decreasing log odds 
respectively. Such a relationship is not surprising. Researchers such as Adair (1991) found that high grades 
in mathematics and science and high SAT scores in mathematics are significantly related to persistence in 
college. 

Although the pre-enrollment factors selected for analysis in this study have a high rate for 
correctly classifying persisters, ongoing study is required to develop a model for retention in college that 
correctly classifies nonpersisters at a higher rate. When this at-risk group begins its college career, 
admissions and counseling personnel would then have more specific interventions to offer. 

The results do support the contention that the "digital divide" is shrinking, particularly for students 
at the community college in this study as reflected by the majority of the respondents who indicate that they 
plan to use a computer for college assignments; are certain of their access to a computer; and have a 
computer at home. In the United States in 1 997, one-half of the people 1 8 years old and older used 
computers, with 47% using one at work, home, and/or school (U.S. Census Bureau, 1999). More 
importantly, the significant relationship occurring between reported current computer use for school or 
work and persistence in the community college among the traditional and nontraditional aged students in 
this study suggests a potential academic benefit from such usage. On the other hand, it may be that current 
computer users at work and school are initially more competent, motivated individuals. Future researchers 
may wish to examine this phenomenon along with continued examination of the multiple roles computers 
may play in student success in college. 
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Abstract 

Self-direction is a necessary characteristic in lifelong learners. Teachers, in order to make best 
use of the new tools and changes in approaches to teaching, must be life-long learners. As 
technology is ever changing, simply providing the pre-service teacher with basic technology skills 
will not be enough. They must be able to self-direct and self-regulate their learning for the 
duration of their careers if they are to effectively practice their profession. A resource-based, 
WWW-based learning environment was developed for a course in teaching methods, lesson 
planning, technology skills, and technology integration strategies for students in a teacher 
education program. Pre- and post-test versions of a technology comfort and expertise inventory 
and the Self-Directed Learning Readiness Scale (Guglielmino, 1977) were administered to as part 
of the course. No significant differences were found in either technology expertise and confidence 
or self-directedness based on the instruments used. Student reflection papers and interviews with 
course instructors illuminate these findings. 



Introduction 

Federal, state, and local agencies are investing billions of dollars to equip schools with the 
technology that may well be the key to improving the learning experience of our nation’s youth. Despite 
these gargantuan investments, only 20 percent of the 2,5 million teachers currently working in public 
schools feel comfortable using these technologies in the classroom (U,S, Department of Education, 1999; 
Technology Counts, 1 997), As reeducating the existing teaching force to take full advantage of these 
technology tools will require expensive professional development over many years, the preparedness of 
new teachers entering the field becomes critical to the success of these investments to improve education. 
As the federal government predicts that 2.2 million new teaches will be needed in the next decade, the time 
to address these issues is now (Milken Exchange on Education Technology, 1999), 

Teacher education programs across the country struggle with how to increase the technology 
integration skills of the students they educate. Not only are standards such as the ISTE National Standards 
for Technology in Teacher Education (International Society for Technology in Education Accreditation 
Committee, 1 998) being adopted, states are now requiring institutions to guarantee the technology 
proficiency of their graduates (University System of Georgia Board of Regents, 1998), At the rate of 
technological innovation, will it make any difference if pre-service teachers are taught simple productivity 
skills for both themselves and their students once the technology changes? What exactly should we be 
teaching pre-service educators about using technology in their future classrooms? The answer to these 
questions might help teacher educators to focus not only on what content and skills need to be taught, but 
also on what instructional strategies might be used in teaching about technology integration. 



Factors Hindering Technology Use in the Classroom 

Several factors have been cited as hindering new teacher use of technology. These include 
inadequate training in the proper technology skills and methods, lack of technology modeling on the part of 
their university faculty, lack of positive technology experience in school settings, and university faculty 
out-of-touch with the technology explosion in schools and how it is effecting teaching practice (Kent & 
McNergnery, 1999; National Council for Accreditation of Teacher Education, 1997; Persichitte, Tharp, & 
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Cafarella, 1997; Office of Technology Assessment, 1995; Byrum & Cashman, 1993). The re-design of 
entire teacher education programs is called for. 

But is it simply skills and knowledge that are lacking? Shoffner (1996) interviewed innovative 
technology using in-service teachers to determine why they were successful. The common factors across 
the identified population were not skills or knowledge, but rather their readiness to tackle challenges, locate 
and learn new knowledge as needed, and overall confidence in trying new things. These innovative teachers 
demonstrated the characteristics typically identified as those of self-directed learners. 

This exploratory study examines an alternative approach to technology education for pre-service 
educators. Pre-service educators were introduced to a resource rich environment, and then given 
opportunities to use, and thus learn from, this environment. By interacting with this environment, it is 
hoped that pre-service educators will learn not only technology and integration skills, but will at the same 
time develop or increase self-directed learning skills which in turn might prepare them for successful life- 
long learning. 

Theoretical Foundations of Self-Directed Learning and Self-Direction 

Several have attempted to define the concept of self- directed learning, as early as the 1920s to the 
present (see Lindeman, 1926; Bryson, 1936; Brookfield, 1986; Merriam & Caffarella, 1991, and others). 
Gerstner (1992) gives a thorough overview of the myriad definitions of self-directed learning, as does 
Caffarella (1993). Perhaps the most widely recognized name in this particular area of adult learning theory 
is Malcolm Knowles, whom is credited with popularizing the concept. Knowles (1975) defined self- 
directed learning (SDL) as follows: 

Self-directed learning describes a process in which individuals take the initiative, with or 
without the help of others, in diagnosing their learning needs, formulating learning goals, 
identifying human and material resources for learning, choosing and implementing 
appropriate learning strategies, and evaluating learning outcomes. 



In contrast, Gerstner (1992) distinguishes between self-directed learning and learner self- 
directness. Viewing the concept as learner self-directedness, Gerstner argues, offers a focusing of the 
concept of SDL, as well as allowing for a broader interpretation, and is the preferable term, as it 
necessitates focus on the learner and suggests a range of forms for its implementation. Long and associates 
(1988) note that there is some experimental evidence of self-directedness acting as a learning style. 
Researchers have established correlations between the level of self-direction in an individual and both the 
quantity of self-directed projects the individual initiates and the amount of time devoted to each project. In 
addition, researchers have found that high scores on the Self-Directed Learning Readiness Scale (SDLRS) 
(Guglielmino, 1977) correspond to facility in planning, organizing, conducting, and evaluating a learning 
project. 

Guglielmino (1977) offered the following characteristics as indicators of the construct of self- 
directedness: initiative; independence; persistence; sense of responsibility for learning; curiosity; ability to 
view problems as challenges; goal orientation; desire to learn or change; and enjoyment from learning. 

Long (1992) proposed that self-directedness is a normal human trait, and that every human demonstrates 
this trait in varying degrees. This supports Guglielmino’s (1977) notion that self-directedness is an attribute 
that can be measured. Long further proposes that interaction of this trait with conditions could 
pathologically manifest self-directedness in learners, meaning that self-directedness can be facilitated 
through learning experiences. 

Guglielmino and Guglielmino (1988) long ago foresaw the impact of the Information Age and the 
accelerating technological revolution on the need for increased self directed learning option in business and 
industry. In anticipation of the move to an information-based society, the Guglielminos proposed the 
creation of learning resource centers that were easily acceptable to all employees. These needs and 
proposed solutions are also applicable in the teaching field, and serve as a foundation for the design of this 
pre-service educational technology course. 



Knowles, 1975 
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Course Design 

IT 3210, Teachers and Technology, is a required course for Middle Childhood and Secondary 
Education students, and an elective course for Early Childhood Education students, and for Foreign 
Language, Art and Music, and Kinesiology/Health students seeking teacher certification. Two years ago, 
the standard skills-based pre-service technology course underwent a major redesign. In the first year, the 
course refocused from a teacher-resource-based, skills-based to a technology-integration-into -the- 
curriculum approach. This refocus was done in part to address a potential cause of low technology adoption 
in pre -service teachers: deficiencies in technology-integration methods (Leggett & Persichitte, 1998). 

In Fall semester 1998, the course took a truly innovative step: while maintaining a lecture/lab 
approach, a WWW-based, resource-based learning environment was introduced as part of the course (Hill, 
1999; Shoffher, 1999). The course, and its related resource laden WWW site, incorporates a problem- 
centered, activity-based approach where the computer applications are anchored in authentic and familiar 
contexts in which teaching and learning occurs (Cognition and Technology Group at Vanderbilt, 1991; 
Vygotsky, 1978). This approach is based on the view of an open learning environment in which learners 
have direct input on the direction of the course based on their needs (Hannafm, 1999; Hannafm, Hall, Land, 
& Hill, 1 994). In navigating through the environment and tackling challenges, it is proposed that students 
will also develop self-directed learning skills, which will serve them well as they enter the teaching 
profession. Along with confidence in using the technology, self-directed learning skills have been identified 
as a characteristic of successful technology-using teachers (Shoffher, 1996). In addition, the course serves 
as an introductory teaching methods course, introducing pre-service students to such concepts as 
instructional objectives, lesson planning, evaluation, and assessment. The pre-service students generate a 
portfolio documenting the design of technology-supported instructional environment that facilitates student 
learning through the design and development of student-centered learning activities. The RBLE can be 
accessed at http://msit.gsu.edu/IT/32 1 0/ 

The site map for the course appears below in Figure 1 . 




Figure L Course Site Map of IT 3210: Teachers and Technolosv 
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Methods 



A resource-based learning environment was developed in Fall 1 998 to serve as the basis of a 
course for pre-service educators as part of the Professional Studies portion of the Middle Childhood 
Education and Secondary Education program of study. Pre- and post-test versions of a technology comfort 
and expertise inventory and the Self-Directed Learning Readiness Scale (form A) (Guglielmino, 1977) 
were administered to as part of the course. 

The technology comfort and expertise inventory is a locally developed self-report instrument 
containing thirteen demographic items and 25 technology comfort and expertise items. Technology comfort 
and expertise items include both open-ended and multiple choice responses, and address skills and attitudes 
toward technology use. 

The Self-Directed Learning Readiness Scale (form A) (Guglielmino, 1977) is a 41 -item 
questionnaire with Likert type responses design to indicate a person’s readiness to engage in self-directed 
learning activities. The SDLRS is the most widely used instrument of this type and was therefore chosen 
for the study. 

The basic factors said to underlie the construct of the SDLRS are as follows: 

1 . An openness to learning opportunities. 

2. Self-concept as an effective learner. 

3. Initiative and independence in learning: tolerance of risk, ambiguity, and complexity in 
learning. 

4. Acceptance of responsibility for one’s own learning. 

5. The love of learning. 

6. Creativity; risk taking, skill in designing atypical solutions, and ability to conceive of 
multiple approaches to topics. 

7. Future orientation; self-perception as a lifelong learner. 

8. Problem-solving skills; skills for study and problem solving (Long, 1991). 

All of the above skills are necessary for teacher success in the technology infused classroom. 

In addition, students were asked reflect on the following questions a the beginning, middle, and 
end of the course: 

1 . What do you hope/expect to learn while you are in this course? /Have your expectations 
for what you hoped to learn in the course changed since you wrote your last Reflection 
Paper? 

2. How much do you feel you have leamed/grown in this course to date? 

3. Do you feel you have gained a better understanding of how to tackle challenges? 

4. What are you doing differently with technology than before you took the course? 

Responses in the reflection papers were examined to gain further, richer insight into how students 
are using the learning environment to address teaching and technology concerns. 

Students completed the technology comfort and expertise inventory and SDLRS as part of the 
course. Students were given an explanation of the study, and students indicated their willingness for their 
data to be included in this exploratory study by affixing their signature to an “informed consent” form. To 
date, data has been collected from 40 students in the Fall 1998 semester and 40 students in the Spring 1999 
semester. 

Results 

Data analyses for this study are ongoing. Data continues to be collected each semester. 

Preliminary results from the first year of the study follow. 
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Technology Comfort and Expertise 

The self-report of technology comfort and expertise results were grouped into three areas: comfort, 
perceived expertise, and perceived software proficiency. Although comparisons between pre- and post- 
course responses show that students increased slightly in all three of these areas, the differences were not 
significant. In contrast, student reflection papers submitted at the close of the semester show that students 
felt they had gained a great deal in all three areas. At the beginning of the semester, perceived themselves 
as knowledgeable about the use of technology in education, and ranked themselves as comfortable in their 
technology knowledge. However, it is apparent from their reflection papers that they had initially 
overestimated their knowledge and comfort. Simply put, the students didn’t know what it was they didn’t 
know. As one student succinctly commented, “I did not expect to learn as much as I did because I did not 
realize there was that much out there.” 

Although many had some technology skills and experience with several software packages, how 
they might actually use these skills in a classroom situation was initially missing. In their final reflection 
paper, however, students felt confident that they could dynamically use technology in their classrooms. For 
example, “...I no longer believe that I have to master a certain type of technology before I let my students 
use it. They can learn with me, which will help them develop problem-solving techniques.” Another 
student commented, “To be honest, I was kind of hesitant about using technology in the classroom because 
what if my students know more than me? Now I understand that even when they do there is nothing wrong 
with them teaching me how to do something. We can all (yes, even the teacher) work as a team to learn!” 

Student comments also indicate an increase in confidence in using the computer. One novice 
student offered, ‘I would just like to thank you for helping eliminate my fear of technology. I am totally 
psyched about learning more...” Many students commented on their increasing confidence with their 
computer skills and the ability to tackle challenges. 

Self-Directedness 

Comparison of pre- and post-test scores on the SDLRS showed no significant difference. In fact, 
the scores decreased slightly, indicating a potential decrease in self-directedness. Two of three items that 
decreased in self-directness scores pre- to post- had to do with traditional classroom instruction, including, 
“In a classroom, I expect the teacher to tell all class members exactly what to do at all times,” and “If I can 
understand something well enough to get a good grade on a test, it doesn’t bother me if I still have 
questions about it.” This is possibly due to student frustration with the amount of ambiguity in the course. 
Student commentary on self-directness gleaned from the reflection papers was mixed. 

It is interesting to note that students perceive themselves to be highly self-directed at the beginning 
of the course. However, informal interviews with course instructors indicate many students outwardly 
display extreme dependence. Many students throughout the course reportedly displayed this dependence on 
traditional teaching methods. Instructors’ perceptions were that many students were empowered greatly 
through by the course design for flexibility, individual pacing, and opportunities for problem solving, and 
that many students increased in self-directedness as technology comfort levels grew. However, course 
instructors found some students very resentful of the nontraditional learning environment. Instructors felt 
students were particularly resentful of having to apply themselves to work in this “fuzzy’ environment. 
Some students expressed resentment at having to attend class sessions (“Why isn’t this class totally 
online?”). 

Factor analysis of SDLRS results is pending. 



Conclusions 

While students were successful in their acquisition of technology expertise and confidence, many 
were seemingly overwhelmed with the volume of course content. While instructors perceived that student 
increased in their self-directedness by the end of the course, some students expressed frustration at the 
nontraditional course approach, including the large mount of course content, and the time and effort it took 
to navigate through the content and to apply the technology and teaching skills in a practical manner. 
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Implications for Further Research and Development 



This was an exploratory study of a small scale. Data continues to be collected. Based on instructor 
and student comments, the course delivery method has been expanded to a true hybrid delivery system: 
students now meet in the classroom, discuss via on-line bulletin boards, and interact with the vast on-line 
resources. It is anticipated that students will continue to use the course resources throughout the remainder 
of their pre-service program, and indeed into their novice teacher years. 

Other planned instructional development includes the creation of “on-line modules,” or Internet- 
based instructional units on each of the teclmologies as well as on technology integration methods and 
theory. It is anticipated that programs that do not have a required technology course will use these modules 
as “Just-In-Time” instruction for students to learn technology skills and methods throughout their pre- 
service coursework. 

While on-line instruction has been successful at the graduate level, little, if any, research has been 
done at the undergraduate level. If the results of this exploratory study are any indication, one must 
question if undergraduate students have the self-directness necessary to successfully complete on-line 
study. 
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The Impact of Information Literacy on Web-based 

Learning 
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Abstract 

As Internet information resources proliferate, schools are implementing Web-based Learning 
(WBL) programs. Without adequate information literacy skills, teachers and students are likely to 
exclude information resources contributing to a solid knowledge base. This paper examines issues 
and problems associated with accessing disparate sources of information. An experiment was 
conducted to identify differences in the type and amount of information retrieved by Internet 
search engines from Lycos, WebCrawler, Alta Vista, Infoseek, Excite, HotBot and the local search 
engine for commercial Dialog databases. The results indicate a diversity of sources with little 
overlap. Relevance assessment is reported as Precision values. A measure of document goodness 
was developed and overlap between searches was calculated. 



Introduction 

The Internet is having a profound impact within the education field in the areas of course delivery 
and information gathering. Web-based instruction holds the potential for elimination of textbooks and the 
integration of on-demand information as a basis for course content Children have the opportunity to 
converse with experts, engage in interactive projects online, and actively participate in building their 
knowledge by following hypermedia links to a wealth of information resources. At the same time the 
success of the children's instruction hinges on the quality of information made available to the student. 



Statement of the Problem 

With the implementation of Web-based Learning (WLB), the quality and quantity of information 
resources will have a significant impact on a student's knowledge base. It is well known that end-users 
seldom search successfully. Thus ease of access will determine what information will contribute the most 
to learning. Because of this, the knowledge base of students will become so fragmented that 
communication and decision making will be impaired. 



Purpose 

It has been estimated that approximately 95% of all recorded knowledge is still not available in 
machine-readable form. As information producers provide Internet access to their products, the problem 
will abate considerably for all but the most serious of scholars. Unfortunately, even though the databases 
are available online, the Internet search engines do not index many publications. Students brought up 
relying on Web documents may not consider extending their searches to large library-based collections, 
such as the Library of Congress, or to the databases of commercial vendors, such as Dialog, which require 
prior authorization and charge a fee. To show the magnitude of these databases, consider that while the 
Web may have 50 million to 100 million pages of information, DIALOG alone has approximately 4 billion 
pages, totaling over 7 terabytes. 

It then becomes important to demonstrate that the type of information retrieved from different 
sources varies both in quality and quantity. The intent is to stimulate educators in making intelligent 
choices in targeting information sources, and to consider the impact on current and future educational 
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teaching strategies. The purpose of this paper is to demonstrate differences in the type and amount of 
information one might retrieve using Internet search engines and compare this to searches in databases not 
indexed by these search engines. 

Rationale 

There is no reason that the Internet should not be the medium of choice for information searching. 
While a great deal of questionable material is also present, the Web holds a wealth of resources particular 
suited for teachers and students alike. However, a dichotomy has developed whereby educators consider 
the Web to be an information source for education, and commercial databases are considered to serve the 
information needs of the business, industrial and research sectors. (Coopers & Lybrand 1 997) Thus many 
useful sources of information go unexamined. 

End-User Searching 

Presently, lack of equipment may be the biggest stumbling block for student access to the Web. A 
recent study by the CEO Forum indicated that only 3% of schools are on target integrating technology with 
the curriculum. (Henry, 1997) 12% come close to the ideal; students use the technology to research, create, 
communicate and practice basic skills. 26 % have a blend of old and new technology, but teachers lack the 
training to fully utilize computers for instruction, much less efficient information searching. The remaining 
59% have older computers that are confined to laboratories or administrative offices. Beyond equipment 
and communication problems, human behavior poses the greatest difficulty in accessing relevant 
information sources. Some human traits include: 

• The Immediacy Factor: No matter how long they have procrastinated, people prefer just- 
in-time information, on their desktop, in useable form. 

• Communication: People avoid direct face-to-face contact with others, such as librarians, 
especially if users are uncertain how to articulate the information need. 

• People prefer not to spend their own money for intangible goods, such as information. 

• People avoid learning new procedures, especially if they appear complicated. 

• Given a number of alternative courses of action, people prefer the easiest method. 

In Web searching, there are three main operational issues as well. (Lockard, 1997). The term 
"wandering” refers to the ability of the user to meander aimlessly through vast resources, potentially 
gaining little beyond enjoyment. A user can also experience "disorientation" with the potential of becoming 
lost in hyperspace and losing track of the original information need. The third problem is that users may 
experience cognitive overload, officially termed Information Fatigue Syndrome. (Reuters, 1996) The user 
develops a sense of inability to cope with and draw meaning from the overwhelming array of information. 
The teacher's role as a facilitator may do little good without adequate training. Training of teachers and 
training of trainers remain a major issue. 

Information Literacy 

A major solution for coping with human weaknesses is the integration of information literacy, and 
in particular electronic literacy, as an integral first step in a Web-based curriculum. Graver, 1 997) The 
following competencies have been described as desirable in order to be information literate: (Doyle, 1994) 

• defining the need for information 

• initiating the search strategy 

• locating the resources 

® accessing and comprehending the information 

• interpreting the information 

• communicating the information 

• evaluating the product and the process 
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With these competencies students and teachers can stay focused, search successfully, and build a 
solid knowledge base. Unfortunately, most Internet workshops get users online quickly and generally 
ignore other elements of information literacy. 

Related Literature 

Beyond the human factor, search engine performance has been of great interest. Numerous studies 
are conducted annually to identify their retrieval effectiveness. (Internet World, 1997) Most of these studies 
use simplistic subjects and give little or no consideration to information resources beyond those indexed by 
the search engines. Some studies have compared database performance against Web searches. 

A study conducted at Moscow State University compared the retrieval effectiveness of eleven 
search engines against traditional technical bibliographic databases. (Lebedev, 1997) The author concluded 
that the maximum number of documents which can be found in the Web is less than 10% of the numbers 
that can be found using a scientific database such as INSPEC or CAS. Interestingly, the multi-threaded 
search engine MetaCrawler retrieved fewer hits than separate search engines. The retrieved documents 
were classified according to type, but relevance assessment was not performed. The conclusion was that the 
Web can find interesting supplementary information on authors, research projects, and foundations 
supporting their work, thus both sources should be used. The study points out one of the nicest features of 
the Web, namely the ability to quickly publish ideas and real data before the information becomes obsolete. 
At the same time, the potential loss of information is made quite clear. 

The largest study to compare the type of information retrieved from commercial and library-based 
databases with Web documents is underway as of this writing, (sef2@comell.edu) Sponsored by Dialog 
and Dow Jones Interactive, up to 200 professional searchers are keeping a log of searches performed on the 
Web version of Dialog or Dow Jones Interactive and on at least one Web search engine. They are 
attempting to cover scientific, business and consumer topics. Up to thirty of the most relevant documents 
will be examined. As with our study, real questions are being used. While it is too early to tell, the 
compilation of aggregate data will be a massive undertaking. The advantage of our study is the ability to 
control variables in an experimental setting. Furthermore, we provide measures of relevance, source 
quality, and duplication across searches. Other studies have analyzed the performance of web search 
engines without comparing them to online databases. Research conducted by the journal Science analyzed 
coverage and recency of web search engines. (Lawrence, 1998) The authors point out that no single web 
search engine indexes more than one third of the web. Coverage is increased significantly by combining 
more than one search engine. While the study examined overlap between search engines, it did not compare 
search precision. A study conducted at Winona State University compared the precision of searches 
between five major web search engines. (Leighton, 1997) The authors used real reference questions and 
analyzed the precision and relevance of results on the first twenty hits. Their analysis showed that Alta 
Vista, Excite, and Infoseek were superior to HotBot and Lycos. 



Methodology 

In the present study one of our concerns was to determine what information is missed by users 
who rely solely on searching WWW sites, and not those offered by online fee-based vendors and libraries, 
Four reference questions which had been asked of information searchers by end users were selected from 
an archived list. An additional question was selected from an inquiry by a university faculty member 
seeking to identify lecture and research material. The intent was to provide a mix of questions involving 
both factual information and complex subject searches. The questions are as follows: 

• How do you measure the effectiveness of corporate sponsorships? 

• What is the impact of corporate culture on information policy? 

• Can you locate the lyrics and music to old German beer drinking songs? 

• I need a biography of Madonna. 

• I need a picture of Chuck Berry doing the “duck walk'\ 
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These questions were translated into the query language of six Internet search engines, and the 
local search engine for Dialog, Insofar as possible, the same query was run across all search engines. 
Searching individual files in libraries would produce somewhat fewer hits but the process and subsequent 
results would be essentially the same as searching Dialog, Dialog also permits searching multiple files, 
using a common search language. This reduced the tedium of the search process. Thus individual free 
databases were not searched. The primary measure of assessing relevance was precision. Precision is a 
well-known measure, indicating which portion of all retrieved documents is considered relevant. There was 
no attempt to assess "recall", which is a measure of relevant documents not retrieved, A sum total of all 
relevant items from each search could provide an estimate of the recall value. An information specialist 
well versed in accessing the Internet and searching library and commercial databases performed the 
searches, A professor who has taught online searching for 25 years corroborated search strategies. 



Results 

It is important to stress that the intent was not to create the impression that Dialog is a better 
information source than directly accessible Web documents. Had we selected questions relating to 
educational curricula, such as science projects, or used a topical channel, the precision for most search 
engines might have been greater. Furthermore, the results of single concept queries that are subsequently 
refined may ultimately result in better precision. What the results do demonstrate is that there is a great 
difference in the number and type of documents retrieved, as well as the type of information. Thus a user 
should give careful consideration to the information need before taking the easiest path. The following 
table presents a summary of major test results. 

Table 7, Search Results 



Search 

Subject 


Dialog 


Alta Vista 


HotBot 


Info 

Seek 


Lycos 


Excite 


Web Crawler 


Overlap 


Information 
policy & 
corporate 
culture 


48 hits 

p=.06 

$32 

20 

min. 


9.6K 

p=.05 

index=2 


69 

p=,03 

index=3 


3,3M 

p=,l 

index= 

3 


0 

p=0 

index= 

I 


I.5K 

p=.I 

index= 

4 


5IK 

p=0 

index= I 


,01 


Beer songs 


1 7 hits 
p=.5 
$9.64 
2 min. 


2.7M 

p=.15 

index=4 


7,9K 

p=,I 

index=4 


I,I5M 

p=0 

index= 

5 


0 

p=0 

index= 

2 


626K 

p=0 

index= 

4 


6.IK 

p=0 

index=2 


,04 


Corporate 

sponsor 


24 hits 
p=l 
$17 
6 min. 


I. IK 
p=.15 
index=3 


IK 

p=.05 

index=2 


3,7K 

p=.02 

index= 

3 


998 

p=,l 

index= 

3 


250 

p=,2 

index= 

4 


290 

p=,l 

index=3 


,03 


Chuck Berry 
& duck walk 


25 hits 
p=.i 
$16 
14 

min. 


9.2K 

p=.01 

indcx=2 


57 

p=.02 

index=3 


3.3M 

p=.l 

index= 

5 


2 

p=0 

index= 

I 


I.28M 

p=,01 

index= 

2 


5,7K 

p=,01 

index=l 


,01 


Madonna 


9 hits 
p=I 
$21 
3 min. 


52 

p=.15 

index=4 


4.7K 
p=.0I 
index= 1 


3.3M 

p=,2 

index= 

3 


250 

p=.02 

index= 

I 


9.7K 

p=.I5 

index= 

3 


8,2K 

p=,03 

indcx=2 


,05 



The top figure in each cell represents the total number of documents references retrieved. The M is 
for millions, and the K is for thousands. Figures over 999 are rounded off 
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Precision 



The second line provides precision estimates, based on relevance assessment of the first twenty 
hits. Each item was given a rating in increments of one-fourth, based on a subjective assessment of 
relevance, i.e. the likelihood that the document would provide the desired information. If there was no 
indication that a document was relevant, it would be scored zero. If there was no question of relevance, the 
document was assigned a value of one. Relevance was based on the type of document, the source, and 
content. Several of the sites judged likely to hold the answer were visited and the relevance assessment 
proved reasonably accurate. The individual ratings for a search were summed and converted to precision 
values. Thus, a precision value of say . 1 indicates that one in ten documents were likely to contribute to the 
answer. Keep in mind that most search engines rank documents, thus precision likely decreases rapidly as 
documents beyond the first twenty are examined. 

Document Goodness 

Since precision values for most searches were low, a goodness index was developed to reflect 
document quality. As the numbers increase, the documents become more substantive and more likely to 
elicit a visit to the site. Information in these documents may not be judged relevant to the immediate need, 
but users can make better decisions regarding subsequent queries or perhaps switching to a different search 
engine. An index value of 1 stands for blind leads, advertisements, jokes, pornography, and so on. A 
category of 2 may contain announcements of seminars, institutional home pages, or references to 
potentially relevant but unattainable sources. Category 3 starts to produce more serious sources but is short 
on immediate information; for example lists of links, bibliographies, and tables of contents. Class 4 
documents may include more informative documents such as abstracts of papers, news items, speeches, and 
student papers. Category five consists of substantive documents such as white papers, journal articles, and 
full text essays. Rather surprisingly, there was a homogeneity in the type of documents retrieved by a given 
search engine. 

Document Type and Availability 

There were notable differences between the types of documents retrieved on the Web and from the 
Dialog databases. Dialog performed quite well for most searches, retrieving manageable numbers of 
relevant documents. These documents also went back further in time than the Web documents. Dialog 
documents consisted primarily of a mix of bibliographic and full-text documents. The goodness index value 
is not reported for Dialog since all searches produced category 4 or 5 documents. There was a welcome 
relief from advertisements, and printing was fast. The biggest benefits to searching these databases may be 
the use of thesauri for indexing, multiple field structure and the high document quality. Several of the 
Dialog hits did not provide full text but document delivery is available for a fee, or many of the documents 
can be obtained from libraries. This is somewhat frustrating to some users, who would likely do without. 
Web publications were more varied, ranging from student home pages, and advertising to substantive 
articles from reliable sources. Some of the Web sites were trivial or under construction, lacked ordering 
information or caused the browser to crash. 

It was interesting to note substantial differences between the number of records retrieved by search 
engines, and how retrievals clustered around a particular document type. For example, for one search 
engine, ten of the first twenty documents were all government reports from Australia. The sites were 
substantial enough, but not relevant to the question. Alta Vista, Info seek and Excite retrieved documents 
with a good probability of containing infonnation. Lycos and WebCrawler returned mostly ephemeral 
items. HotBot was somewhere in-between. Not all search engines weeded out duplicate sites, but they did 
provide the ability to cluster them. It is a pity that the initial abstract does not show how the key search 
terms are used in the document. Dialog does provide this type of information, and in fact highlights the 
search terms. This makes it much easier to make an initial relevance judgment. Costs and search times were 
provided for Dialog, but are not analyzed since a searcher might obtain similar information from a free 
database. Initial search results for the other search engines all returned within five seconds or less. How 
long a person follows links, and waits for print results will vary radically from user to user. Finally, the last 
column provides an estimate of document overlap between the various searches. Assessing this value is 
somewhat cumbersome, but the number reads somewhat like the precision values. That is, a value of .05 
indicates that between any given two searches, there is likely to be a duplication of about five documents in 
a hundred. The figure can be taken as an estimate of document diversity, somewhat similar to the goodness 
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index. There was practically no overlap between Dialog documents and any of the Web searches. Most of 
the search engines retrieved some references that might lead to relevant sources, but in general performed 
rather poorly. As with other studies, there was very little duplication between them. On the good side, this 
reinforces the notion that information can be gathered from many different places. On the down side, how 
will this impact the knowledge and performance of students across subject areas? 

Advanced Searching 

Although the primary query was kept as uniform as possible across search engines, a variety of 
advanced queries were also run for those searches that initially were not productive. These included using 
Boolean operators, truncation, adjacency and near operators, synonyms, and concept searching. One might 
have expected higher precision values for advanced searches. Surprisingly, many of the advanced searches 
produced little or nothing. Part of the explanation may be that the format and content of Web documents do 
not lend themselves to discriminate analysis. Thus the best search strategy might well be to use simple, 
broad terms and then narrow down the subsets with advanced search features. In this way precision maybe 
improved with not much sacrifice of recall. In all likelihood novice users are more likely to follow links or 
reformulate the original query. Accompanying most search screens were suggestions for alternate searches 
and advertising from bookstores such as Amazon.com and Barnes and Noble, 

One could also select subject channels provided by most search engines, for example education, 
sports, and so on, but based on current observations and experience, recall will suffer more than with the 
latter strategy. The opposite strategy holds true for a structured database, such as those provided by Dialog 
or libraries; complex search strings focus in quickly on relevant documents. 



Conclusions 

The research demonstrates that there are alternate paths to information, and that some are more 
productive than others. It is difficult to ascertain how well a novice searcher might have performed, or if 
they would continue to search across various engines. While the number of hits will vary with time, 
individual search engine characteristics, such as how they index and the type of site visited, should remain 
more stable. Unfortunately, most students do not stay with one subject long enough to find out, or for that 
matter, ever learn to fully exploit a search engine's potential. Numerous studies have shown that many 
people use the default search engine on a browser and never consider advanced search techniques. 

It would be nice if one could determine which search engine is the best for a particular type of 
search, and which sources would be the most productive. With a great deal of experience this can be so. 
Rather ironically, one needs to consult a growing number of off-line print sources as the online 
classification of Web resources is rather basic, (Crowley, 1997) In the business world, where time and 
information quality means money, professional librarians provide this type of service. Local libraries often 
provide similar service, but most users do not avail themselves of these services because of cost, time 
constraints, geographic proximity, or a prefer to search on their own. 

New Technology 

A number of newer technologies are available, but without adequate training, will likely go unused 
by novices. For example. Push technology has been suggested as a way of providing Web-based Learning 
for distance education, but indications are that end-users are reluctant to adopt the technology, (Torok, 
1997) For this reason both Netscape and Explorer have de-emphasized this function in their new browsers. 
Ironically, Dialog has included Push in their search engine. Expert systems, intelligent agents and other 
technologies appear too complex for most users. 

There are a number of issues that will always remain, the most prevalent being that of information 
quality. (Cooke, 1998) The solutions are not necessarily going to be provided with better search engines. 
They are about as easy to use as they can get. The training of teachers in search skills is a more likely 
avenue, A fundamental weakness in the educational system is that teachers themselves are not required to 
develop search skills along with teaching skills and subject knowledge. If the Web is to become the 
primary medium for education, then information and electronic competencies must become an integral 
component of educational curricula. Currently, libraries have been the strongest proponents of information 
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literacy and have the most experience with Internet searching. Integrating librarians into instructional 
models for Web-based learning will resolve several of the problems associated with information access. 
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IN Education 

Philip Wilhauk 
University of Kansas 



Abstract 

Virtual reality (VR) holds the potential to provide unprecedented tools for teaching and learning. 
Presently, research relevant to our understanding of educational VR takes place across a wide 
range of diverse disciplines. While previous works have explored individual aspects of the use of 
VR in instruction, no model has yet emerged to unify these disparate elements of virtual reality 
studies into a clearly defined field. 

This paper presents a theoretical model for the study of VR in education. This model defines the 
field in terms of five domains: interface, interaction, narrative, simulation and construction. It is 
hoped this mental construct will serve to organize and stimulate research while helping to 
establish the field as five interrelated areas of academic inquiry. 



The Need for a Theoretical Model 

The developing technology of virtual reality could potentially lead to the creation of powerful new 
tools for teaching and learning while inspiring innovative models of instruction. Advances in the speed and 
storage capacities of affordable computer systems continue to make it increasingly possible to introduce 
three-dimensional naturalistic interfaces into mainstream educational environments. Unfortunately, 
educators have relatively little direct research from which to form a philosophy of the use of virtual reality 
in education. Furthermore, it has been difficult to organize the relevant research that does exist into a 
cohesive field of study. This problem exists for a number of reasons. 

First, virtual reality itself is still quite new and the technology available today is primitive 
compared to probable future advances in the field. The limitations of present-day VR seriously 
compromise both the degree of immersion and the sophistication of interaction available to users of all but 
the most expensive systems. Thus, it is difficult to effectively study the future use of such an immature 
technology. Fortunately, present-day impediments to rich educational VR are beginning to disappear as 
engineers develop new interface systems for use in other fields such as entertainment and medicine. These 
systems will be capable of producing photo-realistic immersive worlds populated with richly interactive 
objects. 

Second, the pace of technological change in VR far outstrips our ability to understand its practical 
and theoretical implications. Hardware and software developers introduce new capabilities far faster than 
the ability of the educational community to process these changes effectively. This “assimilation gap” 
continues to grow and promises to accelerate. A cohesive approach to the study of any educational 
technology is necessary to minimize its impact. 

Third, virtual reality represents a radically new media paradigm. Although it marks the 
convergence of numerous existing media, VR itself is a distinctly unique medium that promises to 
challenge many of our existing concepts of narrative structure, the limits of computer-mediated interaction 
and even perhaps the nature of reality itself. Inasmuch as VR represents a more extreme break from 
traditional media, the effective use of virtual reality in education will doubtlessly require a significant 
reevaluation of many firmly entrenched practices and deeply held concepts of curriculum and instruction. 

Fourth, the study of virtual reality encompasses a diverse range of fields of interest. A 
comprehensive understanding of the use of VR in education will require research across a broad range of 
(sometimes seemingly unrelated) subjects. Even today, work directly relevant to the study of educational 
VR has been completed in such fields as: computer graphics (Sutherland, 1965), learning theory (Bricken, 
1990), cognitive science and perception (Pinker, 1995), human factors, pedagogy, dramatic theory (Laurel, 
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1991), and artificial intelligence. Often, researchers in these fields address problems with very different 
vocabularies and methodologies. 

Perhaps as a result of this disparate foundation, no one has yet to establish a comprehensive 
unified framework for the study of virtual reality in education. A theoretical model would assist greatly in 
pulling together existing relevant work while helping to define and promote the larger field as an area of 
academic pursuit. The structure of this model would serve as a mental construct to coalesce present and 
future research into a cohesive body of knowledge. In this way, the broad problems facing the field can be 
better tackled by bridging the gaps between individual domains. 



History 

Although the term “virtual reality” itself dates only to the late 1980’s, the concept it represents 
actually originated more than two decades earlier. Ivan Sutherland (1965) proposed the concept of a three- 
dimensional graphical computer interface in a pioneering paper entitled “The Ultimate Display.” In recent 
years however, many different interpretations of term “virtual reality” have emerged. This may be due in 
part to the willingness of some to market products as “virtual reality” which in fact had little connection to 
Sutherland’s vision. Most seriously considered definitions of VR, however, feature at least some of the 
components present in his work. 

Attempts have been made to reconcile some of the conflicting 
definitions. To this end, some have suggested that there are actually 
many different kinds of virtual reality. A particularly notable example 
of this was Jacobson's (1994) proposal identifying four classifications 
of VR: immersive, desktop, projection and simulation. 

The differences between almost all of these classifications, 
however, are the products of present-day technical limitations rather 
than unique theoretical constructs. Projection and desktop VR, for 
example, exist only in the void of affordable immersive, first-person 
virtual reality. Indeed, our very concept of the "desktop" computer 
is likely to disappear in coming years (Negroponte, 1995) . For this 
reason, subsequent uses of the term "virtual reality" in this paper 
will refer to its mature form. The ideal of this form can be expressed, 
to paraphrase Sutherland, as "ultimate VR." The virtual reality to 
which I refer includes each of the following: 

1 . The complex simulation of a real or fictional environment 

2. A naturalistic, first-person interface 

3. Meaningfully and naturally interactive elements 

4. Full immersion of the user in the virtual world 

5. A non-linear narrative structure 

One could then measure the sophistication of a virtual reality by the degree to which it satisfies 
each of these requirements in a manner approaching what the user experiences in real life. “Ultimate VR” 
would be able to pass a sort of modified Turing test in which the user is unable to distinguish between the 
“real” and “virtual” worlds. Because no purely computer-generated virtual realities have yet to approach 
this standard, care should be taken to frame existing research in terms of how well its findings can be 
generalized to at least an approximation of this ideal. This is important if we are to build a body of 
knowledge that will be useful to future researchers. Studies tied too closely to a particular desktop platform 
or present-day format will be of dubious value to researchers ten or twenty years from now. 

Much of the early research into the use of virtual reality in teaching and learning took place in 
three areas: the military, space travel and medicine. Several factors worked to create this situation. First, 
these fields were among the very few which possessed the daunting human and financial resources 
necessary to develop and deliver VR at the time. Second, each field already made extensive use of 
simulation-based training especially in the development of highly technical skills. The instructional model 
of simulation-based technical training naturally leant itself to the experiential nature of VR-based 
instruction. Third, the expensive nature of effective mechanical simulator systems made the pursuit of 
cheaper alternatives economically feasible despite the enormous initial expense. 
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Although preliminary research revealed significant advantages in the use of VR in simulation- 
based training environments, major barriers to the use of VR in mainstream K-12 and higher education 
have remained. Rudimentary VR systems continued to be prohibitively expensive for all but a handful of 
institutions and, even v^hen the financial resources existed, fev^ standards or readily available development 
tools existed. Perhaps most significantly, many existing models of instruction did not lend themselves as 
readily to the experiential nature of VR as did simulation-based training. Clearly, nev^ perspectives on 
teaching and learning were required. 

Theorists have looked to existing work in a number of established disciplines in order to form an 
understanding of the nature of VR as an educational medium. To this end, McLellan (1996) identified five 
theoretical perspectives on virtual reality: Ecological Psychology (Gibson, 1986); Computers-as-Theater 
(Laurel, 1991); Spacemaker Design; Constructivist Learning (Bricken, 1990); and Situated Learning 
(McLellan, 1991). Though an important breakthrough, these perspectives represent philosophical 
approaches to the study of virtual reality, or schools of thought. They do not attempt to break down the 
problems facing the field into components for further study. 



A Five Domain Model for the Study of VR 

The model I propose establishes five interrelated domains within the study of virtual reality in 
education. They are: interface, interaction, narrative, simulation and construction. Although the order in 
which these domains are presented does follow a certain logical progression, it is not intended to imply that 
any one domain is more important or critical to the field than another. Rather, it is meant to illustrate that 
each domain confronts a uniquely challenging set of problems which must be tackled if we are to fully 
realize the enormous potential of virtual reality as a medium for teaching and learning. 

Interface 

Interface encompasses the design, development and testing of hardware and software which can be 
employed to place the user within a static virtual environment. A static virtual environment is one which 
the user is able to explore from a multitude of perspectives, but unable to change in any significant way. 
Head-mounted displays (HMD’s) and tactile simulators are both examples of devices which could be used 
in this regard. Any research exploring the relative strengths and weaknesses of a particular interface 
feature or degree of immersion would fall within this domain. 

Most VR research to this point has addressed problems involving interface. Yet this is typical of 
the early history of any medium. The motion picture camera, after all, had to be invented before critics 
could examine cultural themes in cinema or filmmakers could develop new narrative techniques. Before 
1 900, most of the advances in motion pictures were technical innovations where inventors and engineers 
worked to perfect the mechanical system of exposing, developing and projecting film. Only when the 
motion picture became a mature technology did the modem cinema begin to emerge. 

Virtual reality is not yet a mature technology. As mentioned previously, this is a factor greatly 
inhibiting our understanding of the medium. Today, the requisite components of an immersive VR system 
(HMD’s, extremely fast processors, the capacity to store vast amounts of sensory data, etc.) are still 
expensive and cumbersome. Until VR becomes a more mature and accessible technology, direct 
experimentation outside the interface domain will continue to be expensive, limited and difficult. 
Fortunately, this domain is witnessing rapid progress. Until then, we should be reticent in extending results 
obtained in the study of simple VRML worlds to fully-immersive, highly interactive environments. 

Potential areas of inquiry in the study of interface include: 

• The development and testing of new VR hardware 

• The development and testing of new interface software 

• Refinement of existing VR hardware & interface software 

• Identification of the relative strengths & weaknesses of particular interfaces 

• Principles of virtual interface design 

• The physical effects of prolonged exposure to virtual reality 

• Human perception in virtual environments. 
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Interaction 

As one might deduce from the definition of interface, interaction involves the study of those 
features which permit the user to affect change within a virtual environment. These features can be 
categorized in two ways: human-to-human interaction, and human interaction with computer-controlled 
virtual elements. 

Human-to-human interaction studies address the use of virtual reality to enable new forms of 
communication and collaboration among people. Because virtual reality promises to offer unprecedented 
opportunities in this regard, it is critical that we explore this potential. The use of avatars, for example, 
may impact some of the effects of instructional bias or student motivation. 

The study of interactive elements explores the educational use of dynamic, computer-generated 
objects whose conditions change in response to the actions of a user. These elements could range from 
simple objects on one end to highly engaging synthetic characters at the other. Whereas simple interactive 
objects (virtual golf balls, for example) respond to user input according to external laws or principles, 
synthetic characters appear to exhibit some form of internal intelligence. 

Work in the interaction domain is likely to pull from existing work involving cognitive science, 
sociology, pedagogy and artificial intelligence. While the challenges are daunting, the degree to which 
virtual interaction satisfyingly simulates or facilitates the interaction we find in physical reality will go far 
in determining the ultimate potential of VR. 

Several important areas of inquiry exist in the study of virtual interaction including: 

• The study of human to human interaction in virtual environments 

• The role of interaction in instructional design 

• The use of VR in addressing gender, racial and other types of bias in education 

• Cooperative learning in virtual environments 

• The sociological & psychological implications ofVR 

• Design and development of interactive objects and synthetic characters 

• The use of VR by people with special needs 

• The effects of VR on multicultural education. 



Narrative 

Narrative studies explore the use of new storytelling techniques and methods of artistic expression 
enabled by the unique properties of virtual reality. An inherently non-linear and interactive medium, VR 
will require the development of radically new narrative forms and dramatic structures. In this regard, 
existing work in the fields of dramatic theory, interactive fiction and game development may be useful. 
Narrative techniques may be a critical component in the effective use of virtual environments in instruction. 
Potential areas of inquiry in the study of narrative in VR include: 

• The development of new non-linear narrative structures 

• The role of narrative in the design of instructional environments 

• The effects of narrative techniques on student motivation. 

Simulation 

Simulation has existed as a field of study within educational technology for many years. 

However, like the desktop computer, VR offers the opportunity to create educational simulations of 
unprecedented complexity, engagement and realism. Whereas existing computer-based simulations have 
focused primarily on recreating complex systems, virtual reality promises to allow the creation of 
experiential simulators of unparalleled sophistication. One early innovative use of virtual simulation is the 
ScienceSpace project. In one part of this project, students simulated the experience of being a ball in order 
to better understand Newtonian mechanics (Dede, Salzman, & Loflin, 1996). 
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Potential areas of inquiry in the study of simulation include: 

• “Physicaf’ representations of complex abstract systems such as economics and 
science. 

• The use of historical simulations in the social sciences 

• Scenario-based instruction 

• Experiential education and VR. 



Construction 

The final domain in the study of educational VR is construction. As its name suggests, this area 
investigates the use of virtual reality by students in the creation of original environments or the 
modification of existing worlds. This domain is likely to be influenced heavily by the constructionist 
philosophy of Seymour Papert of MIT. Papert (1994) states that students derive educational benefits when 
engaged in the process of constructing products they find personally meaningful. Influenced by the 
constructivist learning theory of Piaget, constructionist applications of virtual reality could offer learners 
the opportunity to take part in the creation of entire worlds. 

Potential areas of inquiry in the study of instruction include: 

• Development and testing of new tools for constructing virtual environments 

• The application of constructionist principles of instruction to educational VR 

• Student-constructed virtual reality as an artistic medium 

Conclusion 

Only when researchers solve many of the critical problems facing educational VR will it become 
possible to begin to employ the technology effectively in instruction. Hopefully, this model will accelerate 
this progress by helping to synthesize existing work, define a research agenda for the discipline and 
stimulate others to explore this growing branch of educational technology. 
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Introduction 

There is now, in the field of mathematics education, a special need for comprehensive guidelines 
for instructors to refer to in designing and implementing discourse-based instruction. This need has arisen 
in the course of a recent reform movement, or paradigm shift, in mathematics instruction. 

In recent years there have been many calls for reform in the field of mathematics education; 
indeed the field may well be undergoing a paradigm shift. Among the most noteworthy and effective events 
in this innovative trend was the proposal by the National Council of Teachers of Mathematics [NCTM] 
(1989) of four new standards for the mathematics curriculum: mathematics as problem solving, 
mathematics as reasoning, mathematics as connection, and mathematics as communication. The purpose of 
these standards was to establish a broad framework to guide the reform process and to improve the quality 
of school mathematics instruction. 

Let us develop each of these principles in a bit more detail: The NCTM version of the new 
paradigm advocates that school mathematics should adopt a problem-solving and mathematical inquiry 
approach, should cultivate the ability to think critically and reason logically, should make connections 
between mathematics, other subject matters, and everyday life, and should encourage discourse both 
between the instructors and the learners and among the learners as they undertake mathematics problem 
solving and inquiry. 

This paradigm shift represents a radical departure from the traditional paradigm of mathematics 
education, which emphasized the conveyance by the instructors to the learners of decontextualized 
fragments of mathematical knowledge, rather than the gaining of knowledge through inquiry; and judged 
mastery in terms of an individual’s mastery of solution procedures for word problems, rather than 
achieving it in terms of the sort of critical and logical thinking that arises through discourse with peers or 
instructors. Many theorists and educators now feel that education according to the traditional paradigm 
cannot enable learners to apply their mathematical knowledge flexibly to the sort of complex tasks and 
issues they are likely to face in today’s complex, in formation -rich society, or to communicate 
mathematically with the rest of the community. 

With the advent of these new standards for the mathematics curriculum, a need has arisen for new 
instructional theories that will help learners to engage in these new kinds of mathematics learning. In 
general, the challenge facing such theories is to design discourse-based instruction for mathematical 
problem solving and mathematical inquiry that will enhance critical and logical thinking, using wherever 
possible authentic problems drawn from across the curriculum and real life situations. 

“Discourse” is a crucial and recurrent term here, designating the sort of classroom activities and 
the mode of instruction proper to new-paradigm mathematics learning. “Discourse refers to the ways of 
representing, thinking, talking, and agreeing and disagreeing that instructors and students used to engage in 
[learning] tasks” (NCTM, 1991, p. 20). Discourse can be understood as a single activity, but the nature of 
its implicit content is double: first as concerning the nature of mathematics knowledge, second as 
concerning the norms of the community members. That is to say, in negotiating what is right and true, it 
also negotiates what counts as right and true (NCTM, 1991). Thus, current trends of discourse-based 
instruction have been built on two groups of instructional methods: 

One involves the setting up of processes for conducting discourse in mathematics inquiry. These 
processes involve posing questions, explaining, verifying, explaining, justifying, and negotiating the 
solution methods (Simon, 1993). Cobb, Wood, & Yackel (1993) have characterized this sort of discourse as 
“talking about and doing mathematics.” The other group involves designing ways to set up effective 
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norms, both social and mathematical, for the discourse processes — norms such as “listen carefully,” “avoid 
overt reprimands and inappropriate language,” “allow learners to make mistakes,” “allow multiple solution 
methods,” and so on. Cobb et al (1993) have characterized this sort of discourse as “talking about talking 
about mathematics.” The system of norms arising through this discourse is also consistent with what 
Bielaczyc and Collins (1999) refer to as the essential principles for the design of effective learning 
community: the respect-for-others principle, the failure-safe principle, the sharing principle, the negotiation 
principle, and so on. 

Many research studies have provided valuable insights and guidelines concerning both groups of 
discourse in mathematical inquiry (e.g., Cobb et al., 1993; Lampert, 1990; Lo & Wheatley, 1994; Hoyles, 
1985). However, most of them focus on individual methods for the discourse-based instruction itself (for 
example, on the use of reflection, justification, and negotiation in mathematical inquiry), but do not make 
any systematic effort to integrate these methods into comprehensive instructional guidelines for instructors 
to refer to in designing discourse instruction. Nor do they provide guidelines on how to integrate these 
individual methods in order to use them together in an actual mathematics classroom environment. 

Therefore, the purpose of this paper is to develop a tentative set of comprehensive guidelines for 
discourse in mathematical inquiry. Such guidelines should address the implementation of discourse 
methods as a whole, including both “talking about and doing mathematics” (which I will term “processes 
for discourse”) and “talking about talking about mathematics” (which I will term “norms for discourse”); 
and they should show how to use these two groups of discourse methods in mathematical inquiry. Such 
guidelines should help mathematics learners to learn more effectively and efficiently through discourse. In 
developing them, I will draw on relevant research studies as well as on my own on-site investigations. In 
order to pull all of these notions together, I will use Reigeluth’s (1999) instructional-design theory as a 
framework for building my instructional guidelines for Discourse in Mathematics Inquiry (“DMI”). 

In the following section, as a preliminary to an instructional-design theory analysis of discourse 
mathematics learning theory, I will briefly describe the components of Reigeluth’s instructional-design 
theory (IDT) framework. 

Analysis of Instructional-Design Theory 

An instructional-design theory is “a theory that offers explicit guidance on how to better help 
people learn and develop” (Reigeluth, 1999). Reigeluth indicates a group of qualities characteristic of all 
instructional-design theories: 

First, instructional-design theories are design-oriented or goal-oriented; they focus on designing 
specific ways to attain given learning or developmental goals. They are prescriptive rather than descriptive 
in nature, in the sense that they offer guidelines as to what method(s) to use to best attain a given goal. 

Second, instructional-design theories comprise (at least) two components: methods and situations. 
Methods are ways to support and facilitate learning or development. Situations are indications as to when 
and when not to use those methods. Thus the methods that instructional design theories offer are 
situational, not universal: they do not ask, “what is the best instructional method?” but “how does a certain 
instructional method apply to a certain learning situation?” 

Third, instructional design theories break down methods of instruction further into distinct 
components. A method is not necessarily regarded as an unbreakable unit, but as a cluster of different ways 
of doing things. Moreover, methods are treated probabilistically, rather than deterministically — as affecting 
the likelihood, rather than necessarily ensuring, the achievement, of learning goals. 

Fourth, instructional design theories treat the instructional situation as consisting of two aspects: 
the conditions under which the instructional design theory takes place, and the desired outcomes. 

Fifth, instructional design theories evaluate instructional outcomes in terms of the levels of 
effectiveness, efficiency, and appeal achieved. 

Sixth, instructional design theories take educational values into account. This characteristic 
follows as a consequence from design theory’s goal orientation and its emphasis on the integration of goals, 
situations, and methods. By contrast, any talk of values in descriptive theories is usually considered 
unscientific. An instructional design theory will take values into account in two ways: in determining what 
goals are to be pursued, and in settling on the criteria to be used for evaluating the methods employed. 

Figure 1, drawn from Reigeluth (1999), diagrams the major components of instructional-design 

theories. 
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Fisure. 1 The components of instructional-desisn theories (from Reiseluth, 1999, p. 9) 

In the following sections of this paper, I will analyze my proposed guidelines for Discourse in 
Mathematical Inquiry (DMI) in the following sequence, derived from Reigeluth’s instructional design 
theory framework: the values of DMI, the instructional conditions for DMI, the desired outcomes for DMI, 
the proper implementation of DMI, and finally the instructional methods for DML 

Analysis of Discourse in Mathematics Inquiry 

Discourse Values 

As Reigeluth says, values help to determine what methods of instruction are appropriate for a 
given set of conditions. Therefore it is important, when using the IDT framework, to make the relevant 
values explicit as a preliminary to developing guidelines for instructional methods. Among the more 
prominent DMI values are: 

• Creating learning environments which foster activities similar to those engaged in within 
the community of mathematicians, such as hypothesis, exploration, conjecture, 
questioning, justification, negotiation, and consensus (Simon, 1993). 

• Honoring both individual and shared cognition, as they arise through communication and 
social interactions in the construction of mathematical meaning and understanding. 

• Fostering the extension of mathematical understanding beyond the “content” level to the 
“problem solving,” “epistemic,” and “inquiry” levels (Perkins, 1992). 

• Developing skills in problem-solving, critical thinking, and discourse. 

• Cultivating norms for positive social interactions in the learning community. 



Instructional Conditions 

Instructional conditions for DMI include: the nature of learning, the nature of learning 
environment, the nature of the learner, and the nature of the instructor. As Reigeluth points out, 
instructional conditions may influence which methods will work best to attain the desired outcomes. Not 
every instructional approach is suitable in every learning situation. Therefore, wc need to specify the kinds 
of educational settings for which we will envision our DMI instructional-design guidelines, and the sort of 
situations in which we have observed its effective implementation. What conditions are suitable for DMI? 
What are the conditions under which DMI has been used effectively? 
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The Nature of Learning 

What is to be learned through DMI? DMI has been applied effectively to content that involves 
higher levels of understanding — to tasks that draw on the “problem solving,” “epistemic,” and “inquiry” 
levels, as defined by Perkins and Simmons (Perkins, 1992). As taught according to the new NCTM 
guidelines, mathematics learning begins to deal with this sort of content quite early on. Indeed, Taiwanese 
schools use discourse mathematics learning as early as the third grade. In the traditional third grade 
classroom a ‘word problem’ is usually treated as an opportunity to learn to apply a fixed procedure, and not 
as a way of initiating understanding on a higher level. In the Taiwanese discourse learning system, a ‘word 
problem’ can become an opportunity for the communal construction of mathematical knowledge. 

However, DMI is not appropriate for lower levels of learning — for memorizing facts or mastering 
the fixed steps of routine procedures. DMI is time-consuming; it is not worthwhile to use it for forms of 
understanding which do not extend much beyond the lowest, “content” level. Yet, we must keep in mind 
that, as Perkins and Simmons point out, “the very spirit and structure of subject matters, fields, and 
disciplines lie at [the higher] levels [of understanding]” (1992, pp. 86). The new NCTM standards of 
mathematics education all emphasize these higher levels. 

Thus, while DMI is time consuming, and not appropriate for all subject matters within elementary 
mathematics, it is still an essential instructional strategy, even for the novice learner. And once learners 
have become accustomed to this way of learning, the strategies of discourse can be used flexibly for a 
broad range of topics, adjusted according to the time available and in response to the learners’ level of 
understanding and degree of experience with these methods. 

The Nature of the Learning Environment 

As we have seen, the new paradigm in mathematics instruction models learners learning on the 
sort of discourse that takes place within the community of mathematicians: hypothesis, exploration, 
conjecture, questioning, justification, negotiation, and consensus. In such an environment, learners are free 
to voice their thoughts, and to participate in directing the collaborative inquiry. Ideally, they should be 
allowed to use as much time as they need in order to pursue their understanding and inquiry, and to use as 
many resources as they want in order to support their learning. For DMI, this type of learning environment 
needs to be fostered and protected by both the instructors and the learners themselves. 

Thus, most traditional instructional settings are not suitable for DMI. Lecture -based instruction 
hinders mutual communication among the learners and between instructor and learner. Traditional class 
schedules, with their relatively short and rigidly limited times of instruction for each subject, do not allow 
the extra time often required for DMI, or the flexibility needed to introduce its new learning strategies. 
Textbook-based and instructor-based learning resources are not likely to be sufficient for broad-ranging 
DMI learning, which should be able to exploit the diverse resources provided by the learners, the 
surrounding community, and both school and local libraries. Overall, the traditional classroom orientation, 
instructor face-to-face with the whole group of learners, does not lend itself to DMI. 

In order to alleviate these problems confronting new-paradigm mathematics instruction, schools 
could adopt high technology, such as computer email systems, and conferencing tools to be used in 
learning environments. Through this technology, DMI can be extended beyond limitations, especially the 
time limitations, set by the traditional classroom; everyone’s opinions and understanding could become 
public to all, and each learning community could have equal access to others’ opinions and could 
participate through technology in inter- and intra- community discourse. 

The Nature of the Learner 

We cannot expect young learners involved in DMI to have had prior experience with discourse in 
the classroom setting. But despite the listening-based nature of the traditional classroom instruction 
paradigm, they should not be regarded as totally unskilled in this sort of learning. Indeed, a basic premise 
of DMI is that this sort of learning works well in the classroom because learners are already using an 
informal version of it in the outside world. So if learners are carefully coached for a period of time by 
instructors who are familiar with DMI, in a sort of cognitive apprenticeship, they will soon be comfortable 
with this “new” way of learning. 

Here, a set of practical guidelines for both “processes for discourse” and “norms for discourse” 
can be extremely useful. If a system of norms and procedures is set up to help learners to operate in this 
new environment, clarifying what new roles, expectations, and responsibilities are required for DMI, 
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learners will soon be able to participate smoothly and effectively in DMI. Thus they will attain a degree of 
what Kamii (1985), following Kant and Piaget, has called “autonomy.” This includes both social autonomy 
(taking responsibility for their conduct), and intellectual autonomy (taking responsibility for their own 
learning) (Cobb et al., 1993), 

The Nature of the Instructor 

What sort of instructors are suitable for DMI? What level of training do they need? What school 
background can be expected? What level of mathematical sophistication? Obviously, this approach will 
work best with instructors who are familiar with the principles of discourse learning, who are 
knowledgeable about and sympathetic with the new paradigm in mathematics instruction and the NCTM 
guidelines, and who have some instruction and practice in the use of DMI in the classroom. 

In my own investigations, I have observed instructors using DMI who have, to some extent, most 
of these ideal characteristics. These third and fourth grade Taiwanese public school teachers have had some 
official training in the discourse approach, which is espoused by the Taiwan Ministry of Education, This 
training covers discourse theory, and some instruction in actual classroom procedures, but does not refer to 
any overall IDT framework, A major goal of this paper is to use the IDT framework to develop a set of 
guidelines for DMI that can be utilized even by instructors relatively untrained in discourse theory. 

Desired Instructional Outcomes 

Desired instructional outcomes for DMI include the following: 

Level of Effectiveness 

Level of Effectiveness is how well the instruction works, as indicated by how well the learning 
goals are achieved, DMI instruction does not stress learner effectiveness in the application of fixed 
procedures, but does expect learners to attain much higher levels of understanding at the “problem 
solving,” “epistemic,” and “inquiry” levels, eventually achieving a degree of learner “autonomy,” 

Level of Efficiency 

Level of Efficiency is how long it takes to reach the desired level of effectiveness, DMI is time- 
consuming, Its aim should be not so much efficiency itself as an increase of efficiency. At the beginning of 
DMI instruction, learners are slow in learning, and need to be with coached constantly by the instructor. 
Gradually they progress toward “autonomy,” Finally they become capable of conducting DMI smoothly 
and quickly without much coaching. 

Level of Appeal 

Level of Appeal is the extent to which the learners enjoy the instruction. Both instructors and 
learners should enjoy learning through DMI. Research reports document this high level of appeal (Chung, 
1997), 

Implementation 

It should be mentioned here that the two groups of instructional guidelines for DMI should be 
integrated in instructional practice. “Processes for discourse” should be conducted in an explicit sequence, 
while the “norms for discourse” are applicable throughout, implicitly embedded within the “processes for 
discourse” strategies. This is because of the impossibility of separating the issues of determining norms for 
mathematics discourse from its integral context of doing mathematics: there can be no way to teach the 
norms separately and decontextualized. In practice, the integrated application of “processes for discourse” 
and “norms for discourse” resembles what has been called “just-in-time instruction” (Thiagarajan, 1993), 
That is to say: in the course of their problem-solving activities learners discover certain deficiencies in their 
understanding that need to be made good if they are to complete their project. As these deficiencies are 
discovered, the instructor can make available, “just-in-time,” various processes and norms for the ongoing 
discourse. 

Moreover, once learners have become accustomed to this way of learning, the “processes for 
discourse” strategies do not have to be implemented in a rigid sequence each time. They can be used 
flexibly in response to the immediate needs of the learners, by focusing on certain critical strategies that the 
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learners have not mastered, or on “cutting edge” strategies for using CMI to solve new problems. The key 
to successful implementation is that the instructor be sensitive about when and for how long to coach the 
learners in the application of the appropriate processes and norms. 

Instructional Methods and Strategies for Discourse in Mathematics Inquiry 

As mentioned earlier, instructional methods and strategies for DMI are divided into two main 
categories, (1) processes for discourse- a set of sequential steps for implementing discourse concerning the 
meaning of mathematics and the ways of doing mathematics; and (2) norms for discourse- guidelines, both 
social and mathematical, for effective discourse in the learning community. However, these two groups of 
methods and strategies for DMI are not themselves panaceas. The level of guidance they provide is vague, 
and not likely benefit instruction. In order to increase the probability of success in instruction, there is a 
need to provide more detailed account of how the two groups of instructional methods can be organized 
and how they should be implemented (Nelson, 1 999). 

The following table gives a detailed list of methods within these two groups (Table 1). These 
instructional strategies will be explained in the following section. 
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Table 1. Instructional Methods for Discourse in Mathematics Inquiry 



Processes for Discourse 

(Talking about and doing mathematics) 

1. Pose a problem-centered authentic 
question. 

2. Verify the problem. 

3. Verbalize the solution methods. 

4. Reflect on the solutions methods. 

5. Interpret the solution methods. 

6. Clarify the solution methods. 

7. Question. 

8. Repeat the original solution methods. 

9. Modify the solution methods. 

10. Justify the solution methods. 

11. Supplement the solution methods. 

12. Provide multiple solution methods 

13. Compare the multiple solution methods. 

14. Negotiate the multiple solution methods. 

15. Form consensus. 



Norms for Discourse 

(Talking about talking about mathematics) 



Social Norms for Discourse 

• Listen to everything that is said, with no 
interruptions. 

• Speak loudly, or quietly, as needed. 

• Avoid overt reprimands and inappropriate 
language. 

• Concentrate in class. 

• Try to understand what others are saying. 

• Try to be understood by others. 

• Respect others’ ideas. 

• Raise hand before speaking. 

• Equalize learners’ opportunities to 
participate. 

• Respond openly to others’ ideas, agreeing, 
disagreeing, or asking for further 
explanation. 

• Co-construct and co-maintain norms by the 
instructor and the learners. 

• Relate all discussion to the problem posed. 



Mathematical Norms for Discourse 

• Allow discourse in mathematics inquiry. 

• Allow learners to help each other. 

• Allow negotiation in mathematics inquiry. 

• Allow “mistakes.” 

• Allow performing understanding by using 
multiple mediated tools. 

• Allow multiple solution methods. 

• Allow low-level solution methods. 
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Processes for Discourse (Talking About and Doing Mathematics) 

This group of methods and strategies for DMI involves a set of sequential steps about discussing 
the meaning of mathematics and the ways of doing mathematics. This set of sequential steps is based on the 
social interaction perspective (Hoyles, 1985) that mathematical understanding can be thought of as the 
ability to: 



• form a view of the mathematical idea; 

• step back and reflect upon it; 

• use it appropriately and flexibly; 

• communicate it effectively to another; 

• reflect on another’s perspective from one’s own framework or challenge and logically 
reject this alternative view. 

Of course this understanding relies on discourse. “Language plays an essential role in the 
formation and expression of mathematical ideas. School children should be encouraged to discuss and 
explain the mathematics which they are doing” (Cockroft Report, 1982). Therefore, I (the author) have 
been able to derive a systematic sequence of discourse steps (under the heading of “processes for 
discourse”) based on the literature of mathematics education and my own on-site investigation. 

When this set of sequential steps is first applied, it is better that they be applied step by step by the 
instructor, who can thus provide guidance to novice learners unfamiliar with this way of discourse. As 
learners become more familiar with this way of discourse, the instructor should give the learners 
ownership, letting them conduct the step-by-step discourse on their own. Later, when the learners are quite 
accustomed to this way of discourse, some steps can be skipped if not needed, according to the discourse 
situation. 

1. Pose a problem-centered question 

The learners’ problem-solving process depends crucially on the way in which the instructor poses 
the problems (Lin & Wu, 1 999). Lin and Wu further indicate that the purpose of posing problems is to 
create an interesting situation that will stimulate the learners to think, then to discuss and justify the 
mathematical concepts that they construct. Thus the posing of problems serves as a trigger for mathematics 
discourse. 

Normally, at the beginning of discourse, the instructor poses a mathematical problem or question 
and asks the learners to solve it. The tasks involved should be problem-centered, interesting to the learners, 
authentic to their everyday life, linked to their previous learning or life experience, and Avithin the learners’ 
zone of proximal development (Vygotsky, 1976). They should not be decontextualized as simply numerical 
computation problems (for example, 2+5=?). A real-world, problem-centered approach cultivates the 
learners’ ability to use mathematical knowledge to solve everyday life problems, and is the only way 
abstract numbers can begin to make sense to young learners. The role of the instructor at this stage is that of 
a “problem poser,” instead of a “problem answerer,” in that he/she only poses problems and guides the 
learners to solve them via discourse, and does not demonstrate any solution methods or answers to the 
learners. It is up to the learners to take responsibility for and assume ownership of the solution. At this 
stage, learners are problem solvers who have to actively construct solution methods via discourse, and not 
solving imitators who merely follow what the teacher or their peers tell them to do. According to Winograd 
(1990), when the problem poser gives only indirect guidance and leaves the problem solver more time for 
independent inquiry, success in problem solving is higher. 

The learners should also “have some experience recognizing and formulating their own problems” 
(NCTM, 1989, p. 138). This is to cultivate the learner’s sense of ownership of the problems: when the 
learners themselves pose the problems, these problems become more authentic and more connected to their 
own real-life problems, and the learners show more interest in discussion and participation (Liu, 1998). 

Liang (1996) suggests five ways in which the problem posing activity can take place: 1 . Instructor 
poses, and learners solve; 2. Instructor and learners co-pose, and learners solve; 3. Learners pose, and 
learners themselves solve; 4. Peers pose, and peers solve; and 5. Learners and their family co-pose and co- 
solve. Thus, a problem that engages the interests and shared knowledge of the community of learners opens 
onto a rich and diverse community learning process. 
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2. Verify the problem 

After posing the problem, the instructor should allow the learners to ask the instructor or other 
learners to repeat or explain the meaning of the problem if they do not understand it. This is to make sure 
that the learners understand what the question is. Fully understanding the problem is the necessary first step 
towards a successful solution. Many times, when learners are not able to solve mathematics problems it is 
because they do not understand what the problems are. Providing opportunities for the learners to ask the 
meaning of the problem from the instructor or the other learners can encourage the learners to fully 
understand the problem before starting to solve it. Thus the learners are less likely to just put all the 
numbers together without fully understanding the problem. In verifying the problem, the instructor should 
use strategies like: “who do not understand the problem”? “Any one needs help to understand what the 
problem is?” The instructor also can ask some learners to verbalize publicly — to explain what the problem 
is about, what it asks. This will give the learners a chance to reflect on their understanding of the problem. 
The instructor should also be sensitive about the facial expressions of the learners, looking for expressions 
of puzzlement, etc. 

3. Verbalize solution methods 

After posing the problem, the instructor should give the learners time to solve it. When they are 
ready to show their solution methods to their classmates, the instructor should encourage the learners to 
verbalize and demonstrate how to solve the problem, instead of just showing the solution methods and the 
answers (Ko, 1 996). He should encourage them to show and verbalize every step they used in arriving at an 
answer. 

In this way learners will develop the ability to use language to describe what they are doing, and to 
let classmates understand where the answers come from. 

4. Reflect on the solution methods 

In verbalizing their solution methods, the learners are also expected to “think aloud” about why 
they choose the ways they do to solve the problem. This can provide a chance for the learners to be aware 
of their own thinking processes in arriving at a solution, and help them to make their thinking transparent, 
and to find the mistakes they have made and the problems they have encountered in arriving at their 
solution methods. As Schon (1983) has pointed out, reflection in action helps people to understand what 
their own problems are. Following the learners’ reflections, the instructor can better understand the nature 
of the learners’ construction of mathematics and their level of understanding, and thus become better able 
to guide their learning. Peers too, are better able to make sense of the problem solver’s attempts (Wood, 
Cobb, & Yackel, 1995). 

5. Interpret the solution methods 

The instructor should also ask another learner to interpret again what the solver has said, in order 
to make sure the rest of the classmates understand everything. This is to check that the listeners’ 
understanding of what was said is compatible with what the solver has said. The constructivist view of 
knowledge underlines the necessity of this step: because everyone’s previous experience is different, 
different people’s understanding of reality is unlikely to be identical. Thus what the learners need to seek is 
not identical knowledge but compatible thought (von Glasersfeld, 1995). And this compatible thought will 
increase the level of communication in the learning community. In the process learners should check 
whether or not their ideas are communicable: whether they understand, and are being understood by, each 
other. If not, both sides (the speaker and the listeners) have to find ways to verbalize again in order to be 
understood, because mathematics learning involves not only knowing how to solve problems but also being 
able to communicate with others (Cobb et al., 1993). Having peers interpret each others’ solution methods 
is also a good way to assure that they listen carefully to what others say. Because problem .solving in the 
discourse community is a matter for all the members, not just for the individual involved, co-listening 
needs to be encouraged in order to maintain the necessary level of communication. 

6. Clarify the solution methods 

If the instructor and the listeners find something strange/questionable about the problem solver’s 
interpretation, the problem solver must clarify his/her approach again until it is clearly understood. This is 
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to ensure that problem solvers take responsibility for what they are saying and let listeners understand what 
they are doing, and do not just finish their solutions without taking responsibility. Clarification of the 
solution methods not only helps the learners to articulate their ideas, but also helps the teacher to assess the 
problem solver’s thinking (Brissenden, 1988; Ko, 1996), 

I . Question 

After the problem solvers show their approaches to the solution, the instructor should encourage 
them to ask if there are any questions about their solution by saying, for instance, “Any questions so far?” 
or “Any other questions?” Questions should be encouraged that stimulate higher thinking, rather than 
involving only memory. Such questions can also raise cognitive conflicts for the problem solvers, making 
them find ways to recheck and reflect again on their solution methods and concepts, and guiding them 
towards new ways to explain, support, or modify their methods of arriving at solutions (Tsai, 1991). 

8. Repeat the original solution methods 

After being questioned, the problem solvers are in a position to recognize the possible strengths 
and weakness of their approach. They can continue to insist on the validity of their approach by repeating 
the original solution. Or they can admit that they are doing wrong and alter their solution methods. In any 
case, they should explain their stands clearly. 

9. Modify the solution methods 

If the solutions are not appropriate, the problem solvers should be given opportunities to modify 
their solutions. Also, if they are not able to modify the solutions by themselves, they should be allowed to 
accept help and suggestions from the peers and the instructor. Indeed, when the modification process takes 
place among the peers, it can often have better results than when it involves the instructor, 

10. Justify the solution methods 

If the problem solvers want to insist on their solution, that is all right as long as they justify it for 
themselves. In order to do so, they have to provide complete reasons and proofs in order to persuade the 
peers and the instructor. At this time, peers are encouraged to add support for the problem solvers’ point of 
view, or to provide counter-arguments against the problem solvers’ justification. If they are able to provide 
sound reasons, their solution methods should be considered tentatively valid. Through such a process of 
negotiation among different solutions by the problem solvers and by the listeners, the learners can become 
more aware of their thinking, and can reconstruct or expand it while incorporating different ideas. 

II. Supplement the solution methods 

Sometimes the problem solver may be limited in some aspects of mathematical knowledge and 
can only solve the problems in a limited or incomplete manner. Here the peers can help to supplement the 
given methods with additional insights, information, or solution methods (Tsai, 1991). The instructor or the 
problem solvers can encourage such assistance by asking, “Anything else to supplement?” or, “Who else 
wants to supplement what I have done?” 

12. Provide multiple solution methods 

When the problem solver encounters limitations, the peers should also be encouraged to present 
other, different solution methods. With their different backgrounds and experiences in mathematics 
learning, learners commonly use different solution methods, some primitive and some more advanced. All 
of these are welcomed for presentation to the classmates, so that all can judge the viability of different 
solution methods. The instructor or the problem solver may ask, “Is this the only method?” “Any other 
solution methods?” or, “Anyone have another solution method?” In the mathematics learning community, 
multiple-perspectives on solution methods should be encouraged. Allowing the learners to provide multiple 
solution methods rather than fixing on a single orthodox method will increase their confidence in looking 
for solutions (Lin & Wu, 1 999). 
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13. Compare the multiple solution methods 

It is not enough just to have different methods presented in front of the class. The purpose of the 
presentation of different solution methods is that with different solution methods presented openly in class, 
the learners are given a chance to compare different solutions, and to determine which are appropriate and 
which inappropriate, which efficient and which inefficient. The instructor and the learners may say, *‘We 
have several solution methods now; let’s compare them.” 

14. Negotiate the multiple solution methods 

The everyday life of adults is full of complex problems and different opinions that require 
“negotiation” in order to find viable solutions. Therefore, negotiation is in the very nature of society. 
However, in the world of knowledge, we seldom hear about it, because in the traditional education system, 
knowledge is regarded an eternal truth just sitting there, and the mission of the instructor is to transfer this 
eternal truth exactly to the learner. So in this system there is little room for negotiation in the acquisition of 
knowledge. But things are changing. More and more educators are coming to agree that mathematics 
involves negotiation. Negotiation works in mathematics learning, especially in mathematics inquiry. In 
other words, mathematics knowledge results from negotiation (Tzeng, 1997). 

Thus, after multiple solutions are compared openly in class, the learners are encouraged to discuss 
and negotiate which methods are most viable for arriving at solutions. They need to support their choices 
with reasons, and to settle on the ones they feel most viable for communication. The negotiations here 
might start this way: “If we use this solution method, then this will be the result, because . . .; but if we use 
that solution method, we will reach this different result. . . . Let’s negotiate this.” 

15. Form consensus 

The negotiations should end with a consensus. Otherwise the solution methods cannot be taken as 
valid. The learners are encouraged to decide together which methods are viable for communication. Such 
exchanges could function as a mutual method for establishing communication among the community of 
classmates. For example, a learner might say, “The reason I think this method is most appropriate is 
because it is the most efficient way to the solution. The rest of the solution methods all got the answer right, 
but they needed more time, more different procedures in order to get the answer.” 

Norms for Discourse (Talking about talking about mathematics) 

“Norms for discourse,” or “talking about talking about mathematics” (Cobb et al., 1993), is the 
aspect of mathematics discourse that deals with the construction of norms in the discourse community. 
Inasmuch as mathematics amounts to a social activity, and in order to keep the discourse goes on smoothly, 
there is a need to regulate the norms that provide for and build a common means of communication. 
According to Lo and Wheatley (1994), the norms in the classroom influenced learners’ mathematics 
learning. For example, negative emotional responses would constrain discourse. Accordingly, there was a 
need for the instructor actively to guide norms in order to keep the discourse effective. 

The establishment of norms can be grouped into two aspects: social norms for discourse, and 
mathematical norms for discourse. The social norms deal with the standards of common social courtesy 
needed in order to keep a good classroom atmosphere; without these, the processes of discourse would be 
slow and cumbersome. The mathematical norms deal with the nature of mathematical knowledge, the 
proper attitudes towards learning mathematics (making mistakes is allowed; you can learn mathematics 
through making mistakes), and so on. Both groups of norms cannot be taught decontextualized, but must be 
taught as embedded in authentic situations, through a kind of “just*in*time norming.” 

Social Norms for Discourse 

• Listen to everything that is said, with no interruptions 

This rule is very important for the establishment of discourse. If learners do not listen carefully to 
the end, or interfere with other listeners by interrupting the speakers, and thus do not fully understand what 
others have said, they will not know how to respond appropriately. Unable to respond, they will not be able 
to participate in the discourse processes. As Hoyles (1985) points out, “Listening for learning is not 
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passive; it is an active attempt to incorporate another’s scheme into one’s own; it stimulates one to go 
outside oneself and look again at the arguments” (p. 207). This norm is also a matter of common sense: 
discourse does not mean that people simply say what they want to say and behave as they want to behave; 
and attentive listening is generally regarded as a mark of respect for the speaker. 

To encourage learners to listen carefully, the instructor should remind noise-makers and those who 
seemed absent-minded to listen carefully, or ask them to repeat the statement or to participate immediately 
in the discussion. 

• Speak loudly, or quietly, as needed 

In the discourse environment, it is inevitable that with everybody participating at once in discourse 
and argumentation, they try to speak as loud as possible in an attempt be heard and win the argumentation 
(Tzeng, 1997). Therefore, there is a need to set the rule: the instructor should guide the learners not to shout 
loudly in order to get attention or be heard, and to adjust their voices for different situations. For example, 
in a small group discussion, one’s voice should be heard only within the group; but when speaking to the 
whole class, one’s voice should be heard by the whole class. If not, the instructor as well as the peers might 
say this to the speaker, “Xxx, would you please speak a little louder, so we can hear you” or, “Attention! 
Your discussions are too loud and noisy; this will affect the other groups. Please lower your voices.” The 
value of this norm extends beyond the cultivation of learning; it is also, of course, a basic communication 
skill. 

• Avoid overt repriinands and inappropriate languaee 

“Discourse” has an embedded principle of “co-trust.” So the purpose of discourse is to exchange 
ideas, no matter if they are right or wrong, rather than engage in personal attacks that might suppress 
expression or erode the trust and openness essential to collaborative progress. Therefore, the instructor 
should guide the learners not to use overt reprimands or inappropriate language, even when others made 
mistakes. 

• Concentrate in class 

In the traditional classroom the instructor does most of the talking, while the learners listen 
quietly. Since they are not often called on to participate, it is hard to tell whether or not they are grasping 
the material or, indeed, are even listening. All of the instructors can demand is that they sit there quietly; 
such learners have been described as “the guests in the classroom” (Huang, 1996). Collaborative discourse, 
however, is a matter for all the learners in the classroom. Because all learners must participate in the 
discussion, all must concentrate actively. Of course, concentration is important to any sort of education. But 
in collaborative learning it can also be implemented as a norm of classroom behavior. Where all the 
learners share the tasks of discourse, no one can escape from participating in talking. 

• Try to understand what others are saving 

In discourse, you can participate in the learning process only if you try to understand what others 
are saying. The discourse processes initiated by the instructor should provide learners with many 
opportunities to test and express their understanding of what others are saying. The instructor should guide 
them to use essential elements of the learning process — questioning, and asking for clarification and 
interpretation — to try to understand what others are saying. 

• Try to be understood by others 

Where listeners try actively to understand, speakers also try to be understood. In the classroom, 
when both listeners and speakers strive for understanding, the communication process becomes smoother 
(Hsu, 1999). The discourse processes initiated by the instructor should provide opportunities for speakers 
to reflect on, interpret, clarify, and justify their solution methods. The instructor should help the learners to 
use these discourse methods and strategies in order to make themselves understood by others. 

• Respect others’ ideas 

Respect for others’ ideas is a basic principle in any multi-voiced society. Since collaborative 
learning treats mathematical activity as “community activity-a community project” (Cobb et al., 1993), 
this principle of mutual respect applies to it with the same force. When respect for others’ ideas is 
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established as a principle, learners are not afraid to speak out about their thinking, and do not worry about 
being reprimanded. This brings about an increase in positive interaction and communication among 
learners and between the instructor and the learners (Hsu, 1999; Ko, 1996). So the instructor should guide 
learners to have respect for any idea expressed by any other peers, and made it clear in action that it was 
not only the learners, but the instructor too, who should respect others' ideas 

• Raise hand before speaking 

This is a common form of courtesy, and it is especially helpful in the classroom situation as a way 
of deciding who can speak first. For example, when learners do not raise their hands but just speak out 
loudly, the instructor or peers might say to them, “That did not count, because you did not raise your hand 
before speaking.” The instructor can also encourage this behavior by praising learners who raise their 
hands. 

• Equalize learners* opportunities to participate 

The instructor should be careful to equalize learners’ opportunities to participate in classroom 
discourse. In a traditional classroom, it is the students with higher academic achievements who tend to 
receive the most opportunities to participate in discussion and to help determine the classroom activities. 

As a result, the better perform better, and the poor worse. So, in order to guide learners towards equal 
participation, an instructor might use small groups of four to five learners as units for discourse, as a way of 
providing learners with more opportunities for active participation within the group. In whole-class 
discussions, the instructor should work carefully to assure each learner a chance to speak in the course of 
each unit of instruction. 

• Respond openly to others* ideas, agreeine, disagreeing, or asking for further explanation 

The CMI processes, require open responses. Without these, communicative exchange weakens, 
negotiation cannot function, and true consensus cannot form. Many children have been raised in such a way 
that they find it difficult to express their opinions openly. Therefore, the CMI instructor should work 
constantly to initiate and guide the adoption of the norms of open responsiveness. Only through open 
exchange can your thinking become easily communicated and transparent to others. Learners should be 
encouraged to explicitly express their agreement or disagreement explicitly by saying, “ I agree with your 
solution methods” or, “I do not agree with your solution methods.” 

• Co-construct and co-maintain norms by the instructor and the learners 

These norms can function effectively only when co-constructed and co-maintained by the 
instructor and the learners together. Otherwise, the mathematics inquiry would become chaos and the 
discourse could not proceed. 

• Relate all discussion to the problem posed 

A basic difficulty shared by all young learners becomes especially evident in DMI learning 
situations. In particular, when small groups of learners are not under the instructor’s direct supervision, 
their discussion tends to stray towards topics outside the prescribed classroom activities. Therefore, the 
instructor should remind the learners that their discourse should be related to the problem posed. 



Mathematical Norms for Discourse 



• Allow discourse in mathematics inquiry 

Engagement in the discourse processes shows learners that mathematical learning is an effect of 
discourse, not merely an event that happens within the individual. It is the same processes through which 
mathematicians create new knowledge by openly subjecting mathematical definitions and assumptions to 
constant revision. Therefore, the instructor should allow discourse in mathematics inquiry. 

• Allow learners to help each other 

“Learning is not a matter solely of individual striving, but a journey of cooperative discovery. The 
process of learning entails peers working together to share the joy, the breakthroughs, and the happiness of 
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learning; only then is the learning complete’’ (Lin & Wu, 1999, p. 131). This characterization of learning in 
general applies with equal force to mathematics learning in particular. 

This norm is in opposition to the principles of traditional instruction, where helping others to do 
mathematics is considered cheating. The instructor should actively support for this norm by encouraging 
learners to help others by interpreting, supplementing, and justifying their solution methods for them. 

• Allow negotiation in mathematics inquiry 

Mathematics, like other social learning activities, requires that learners confer with one another so 
as to arrive at the settlement of the matter at hand. The instructor should not regard it as just an activity 
occurring in the head of the individual learner. In traditional mathematics instruction, most learners learn in 
order to perform well on tests, and not in order to solve real-life problems. When a CMI instructor poses 
mathematics problems in terms of real-life situations, learners begin to see how mathematics is related to 
real-life problem solving. 

• Allow ^^mistakes” 

Many learners do not dare to speak out, because they are afraid of being caught in a mistake. In 
this they may be reflecting the adult notion that before you speak out in public you should be sure that your 
ideas are correct. But this mentality is not a good one for learning mathematics. The instructor should leave 
the learner free to guess. Learning from mistakes is very natural in life, and it need not be an 
embarrassment for the learner. The instructor should remind the learners, “It is all right to make mistakes; 
there is no embarrassment or shame in this.” Doing mathematics involves no such thing as absolutely right 
or absolutely wrong; but mathematical knowledge is tentative and constantly seeks viability in the 
community of mathematicians. So the instructor and the learners should not say “absolutely wrong” or 
“absolutely right,” but speak about what is understandable to them or “acceptable to the current 
community.” 

• Allow performing understanding bv usin£ multiple mediated tools 

In the CMI processes, verbalized solution methods may not always be understood, or 
understandable. Such verbalizations may not provide the instructor with satisfactory proof that the learner 
understands the concept. In this sort of situation, the instructor ought to allow the learners to perform their 
understanding by using other mediated tools that they select for themselves. As Gardner has pointed out, 
human understanding is not limited to verbal and numerical operations, but includes musical, spatial, 
interpersonal, and many other aspects. This does not mean that mathematical tools are not important, but 
learners should be temporarily allowed to use the ones they are most comfortable with to express their 
mathematical understanding. But the use of mathematical tools and discourse is the final goal. 

• Allow multiple solution methods 

Every learner’s previous experience being different, all learners construct their understanding in 
different ways. Hence, learners’ solution methods are always likely to be diverse. Therefore, the instructor 
should allow multiple solution methods to be presented in class. With multiple solution methods presented, 
there is often a good opportunity for further discourse involving comparison, negotiation and consensus- 
forming on solution methods. Thus multiple solution methods were welcomed for public presentation, 

0 Allow low-level solution methods 

Constructivist learning theory reminds us that inasmuch as everyone’s experience is different. 
People inevitably construct their understanding in different ways. The CMI instructor should tolerate 
different levels of understanding among the learners. However, this does not mean that the instructor 
should allow learners to stick to low-level solution methods forever. Learners using low-level solution 
methods should be expected to acquire higher-level solution methods through participation in the CMI 
activities of comparing, negotiating and consensus. Ultimately, it is the higher-level and efficient solution 
methods that are most easily understood and accepted by the community. 
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Conclusion 



The purpose of this paper is to develop a set of systematic guidelines for instructional methods and 
strategies to be used in discourse in mathematics inquiry (DMI). In this field, the advent of a new paradigm 
for mathematics instruction has given rise to a pressing need for practical and inclusive instructional 
guidelines. We begin by exploring the new paradigm of mathematics education: problem-centered, 
discourse based approach, aiming to promote critical and logical thinking in the learners, and to enable 
them to solve real-world problems. This new theoretical paradigm calls for a corresponding instructional 
design theory to guide its implementation. 

In respond to this need, we develop a set of guidelines for discourse in mathematics inquiry 
(DMI), framing in my analysis and presentation in terms of Reigeluth’s instructional-design theory model. 
Thus, having laid out, following Reigeluth’s model, the main components of an instructional design theory, 

I proceed to treat the instructional conditions, the desired outcomes, the instructional methods, and the 
underlying instructional values of DMI. 

The proposed guidelines belong to the category ‘instructional methods”. Following Cobb et al. 
(1993), I divide instructional methods and strategies into two main categories: processes for discourse 
(talking about and doing mathematics) and norms for discourse (talking about talking about mathematics). 
Guidelines concerning processes for discourse deal with how, in an actual learning environment, discourse 
concerning the meaning of mathematics and the ways of doing mathematics. These guidelines take the 
form of a set of sequential steps that may be applied as needed to the discourse processes. But the 
implementation of these steps also relies on the establishment of a set of norms for discourse in the 
community of learners. These norms may be categorized into two sub-categories: social norms and 
mathematical norms. In practice, they are just as important as the processes discourse to the successful 
implementation of DMI. 

The use of an instructional design theory framework highlights the multi-dimensional nature of 
any successful implementation of DMI. Guidelines for instructional methods are not panaceas; they are 
applicable only in the context of a clearly defined type of instructional situation, along with its particular 
set of values, goals, etc. In the case of DMI, we find the instructor acting as guide, coach, catalyst, and 
learning resource. Through the instructor, the learners undertake an “apprenticeship” in discourse-based 
mathematics learning, and gradually progressing towards autonomy (Kamii, 1985) as learners. Along the 
way, learners play an active role in constructing mathematical meaning and understanding, take 
responsibility for their own learning, and learn to participate in a discourse community. 
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Abstract. 

Many research studies and evaluations of Web-based instruction are based upon self-reported 
data, which may be highly inaccurate due to faulty memory and other psychological factors. This 
paper discusses four ways to estimate the reliability of self-reported data. The four approaches 
are: Kappa coefficient, Index of Inconsistency, Repeated measures, and correlational/regression 
analysis. 

For research on Web-based instruction, web usage data may be obtained by parsing the user access 
log, setting cookies, or uploading the cache. However, these options may have limited applicability. For 
example, the user access log cannot track users who follow links to other websites. Further, cookie or cache 
approaches may raise privacy issues. In these situations, self-reported data collected by surveys are used. 
This gives rise to the question: How accurate are self-reported data? Cook and Campbell (1979) have 
pointed out that subjects (a) tend to report what they believe the researcher expects to see, or (b) report 
what reflects positively on their own abilities, knowledge, beliefs, or opinions. Another concern about such 
data centers on whether subjects are able to accurately recall past behaviors. Cognitive psychologists have 
warned that the human memory is fallible (Schacter, 1999) and thus the reliability of self-reported data is 
tenuous. 

Although statistical software packages are capable of calculating numbers up to 1 6-32 decimals, 
this precision is meaningless if the data cannot be accurate at even the integer level. Quite a few scholars 
had warned researchers how measurement error could cripple statistical analysis (Blalock, 1 974) and 
suggested that good research practice requires the examination of the quality of the data collected (Fetter, 
Stowe, & Owings, 1984). 

Bias and Variance 

Measurement errors include two components, namely, bias and variable error. Bias is a systematic 
error that tends to push the reported scores toward one extreme end. For example, several versions of IQ 
tests are found to be bias against non-Whites. It means that blacks ands Hispanics tend to receive lower 
scores regardless of their actual intelligence. A variable error, also known as variance, tends to be random. 

In other words, the reported scores could be either above or below the actual scores (Salvucci, Walter, 
Conley, Fink, & Saba, 1997). 

The findings of these two types of measurement errors have different implications. For example, 
in a study comparing self-reported data of height and weight with direct measured data (Hart & Tomazic, 
1999), it was found that subjects tend to over-report their height but under-report their weight. Obviously, 
this kind of error pattern is bias rather than variance. A possible explanation of this bias is that most people 
want to present a better physical image to others. However, if the measurement error is random, the 
explanation may be more complicated. 

One may argue that variable errors, which are random in nature, would cancel out each other and 
thus may not be a threat to the study. For example, the first user may over-estimate his Internet activities by 
1 0%, but the second user may under-estimate hers by 1 0%. In this case, the mean might still be correct. 
However, over-estimation and under-estimation increases variability of the distribution. In many parametric 
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tests, the within-group variability is used as the error term. An inflated variability would definitely affect 
the significance of the test. Some texts may reinforce the above misconception. For example, Deese (1972) 
said. 



Statistical heory tells us that the reliability of observations is proportional to the 
square root of their number. The more observations there are, the more random 
influences there will be. And statistical theory holds that the more random errors 
there are, the more they are likely to cancel one another and produce a normal 
distribution (p.55). 

First, it is true that as the sample size increases the variance of the distribution decreases, it does 
not guarantee that the shape of distribution would approach normality. Second, reliability (the quality of 
data) should be tied to measurement rather than sample size determination. A large sample size with a lot of 
measurement errors, even random errors, would inflate the error term for parametric tests. 

After calculating the standardized difference between the two measurements, a stem-and-leaf plot 
or a histogram can be used to visually examine whether a measurement error is due to systematic bias or 
random variance. 



Remedies 

In spite of the threat of data inaccuracy, it is impossible for the researcher to follow every subject 
with a camcorder and record everything they do. Nonetheless, the researcher can use a subset of subjects to 
obtain observed data such as user log access or daily hardcopy log of web access. The results would then be 
compared to the outcome of all subjects’ self-reported data for an estimation of measurement error. For 
example, when the user access log is available to the researcher, he can ask the subjects to report the 
frequency of their access to the web server. The subjects should not be informed that their Internet 
activities have been logged by the Webmaster as this may affect participant behavior. Also, the researcher 
can ask a subset of users to keep a logbook of their Internet activities for a month. 

Someone may argue that the log book approach is too demanding. Indeed, in many scientific 
research studies, subjects are asked for much more than that. For instance, when scientists studied how 
deep sleep during long range space travel would affect human health, participants were asked to lie in bed 
for a month. In a study concerning how a closed environment affects human psychology during space 
travel, subjects were locked in a room individually for a month. It takes a high cost to seek out scientific 
truths. 

After different sources of data are collected, the discrepancy between the log and the self-reported 
data can be analyzed to estimate the data reliability. At first glance, this approach looks like a test-retest 
reliability, but it isn't. First, in test-retest reliability the instrument used in two or more situations should be 
the same. Second, when the test-retest reliability is low, the source of errors is within the instrument. 
However, when the source of errors is external to the instrument such as human errors, inter-rater reliability 
is more appropriate. 

The above suggested procedure can be conceptualized as a measurement of inter-data reliability, 
which resembles that of inter-rater reliability and repeated measures. There are four ways to estimate the 
inter-rater reliability, namely. Kappa coefficient. Index of Inconsistency, repeated measures ANOVA, and 
regression analysis. The following section describes how these inter-rater reliability measurements may be 
used as inter-data reliability measurements. 

Kappa coefficient 

In psychological and educational research, it is not unusual to employ two or more raters in the 
measurement process when the assessment involves subjective judgments (e.g. grading essays). The inter- 
rater reliability, which is measured by Kappa coefficient, is used to indicate the reliability of the data. For 
example, the performance of the participants is graded by two or more raters as "master" or "non-master" (1 
or 0). Thus, this measurement is usually computed in categorical data analysis procedures such as PROC 
FREQ in SAS and "measurement of agreement" in SPSS’s StatXact. 
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It is important to note that even if 60 percent of two datasets concur with each other, it doesn’t 
mean that the measurements are reliable. Since the outcome is dichotomous, there is a 50 percent chance 
that the two measurements agree. Kappa coefficient takes this into account and demands a higher degree of 
matching to reach consistency. 

In the context of Web-based instruction, each category of self-reported Website usage can be re- 
coded as a binary variable. For example, when question one is "how often do you use telnet," the possible 
categorical responses are "a: daily," "b: three to five times per week," "c: three-five times per month," "d: 
rarely," and "e: never." In this case, the five categories can be re-coded into five variables: QIA, QIB, 
QIC, Q ID, and QIE. Then all these binary variables can be appended to form a R X 2 table as shown in 
the following table. With this data structure, responses can be coded as "1" or "0" and thus measurement of 
classification agreement is possible. The agreement can be computed using Kappa coefficient and thereby 
the reliability of the data may be estimated. 

Index of Inconsistency 

Another way to compute the aforementioned categorical data is Index of Inconsistency (lOI). In 
the above example, because there are two measurements (log and self-reported data) and five options in the 
answer, a 4 X 4 table is formed. The first step to compute lOI is to divide the RXC table into several 2X2 
sub-tables. For example, the last option "never" is treated as one category and all the rest are collapsed into 
another category as "not never," as shown in the following table. 

Table!. 2X2 table for lOL 





Never 


Not never 


Total 


Never 


a 


b 


a+b 


Not Never 


c 


d 


c+d 


Total 


a+c 


b+d 


n=sum(a-d) 



The percent of lOI is computed by the following formula: 

101% = 100*(b+c)/[(2np(l -p)] where p = (a+c)/n 

After the lOI is calculated for each 2X2 sub-table, an average of all indices is used as an indicator 
of the inconsistency of the measure. The criterion to judge whether the data are consistent is as follows: 

• An lOI of less than 20 is low variance; 

• An lOI between 20 and 50 is moderate variance; 

• An lOI above 50 is high variance 

The reliability of the data is expressed in this equation: r = 1 - lOI. Put it simply, reliability is the 
information without the inconsistent portion. 

Repeated measures 

The measurement of inter-data reliability can be conceptualized and proceduralized as a repeated 
measures ANOVA. In a repeated measures ANOVA, measurements are given to the same subjects several 
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times such as pretest, midterm and posttest. In this context, the subjects are also measured repeatedly by the 
web user log, the log book and the self-reported survey. The following is the SAS code for a repeated 
measures ANOVA: 

data one; input user $ webjog log_book self_report; 
cards; 



1 


215 


260 


200 


2 


178 


200 


150 


3 


100 


111 


120 


4 


135 


172 


100 


5 


139 


150 


140 


6 


198 


200 


230 


7 


135 


150 


180 


8 


120 


110 


100 


9 


289 


276 


300 



proc glm; 

classes user; 

model web_log log_book self_report = user; 
repeated time 3; 

run; 

In the above program, the number of visited Websites by nine volunteers is recorded in the user 
access log, the personal log book, and the self-reported survey. The users are treated as a between-subject 
factor while the three measures are regarded as between-measure factor. Table 2 is a condensed output: 



Table 2. Output of repeated measures for inter-data reliability. 



Source of variation 


df 


Mean square 


Between-subject (user) 


8 


10442.50 


Between-measure (time) 


2 


488.93 


Residual 


16 


454.80 



Based on the above information, the reliability coefficient can be calculated using the following 
formula (Fisher, 1946; Horst, 1949): 



r (MS between-i 



- MS residual)/(MS 

between-measure (^^ between-people 






Correlational and regression analysis 

Correlational analysis, which utilizes Pearson’s Product Moment coefficient, is very simple and 
especially useful when the scales of two measurements are not the same. For example, the web server log 
may track the number of pages accesses while the self-reported data are likert-scaled (e.g. How often do 
you browse the Internet? 5=very often, 4=often, 3=sometimes, 2=seldom, 5=never). In this case, the self- 
reported scores can be used as a predictor to regress against page access. 

A similar approach is regression analysis, in which one set of scores (e.g. survey data) is treated as 
the predictor while another set of scores (e.g. user daily log) is considered the dependent variable. If more 
than two measures are employed, a multiple regression model can be applied i.e. the one that yields more 
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accurate result (e.g. Web user access log) is regarded as the dependent variable and all other measures (e,g, 
user daily log, survey data) are treated as independent variables. 

Method 

A preliminary pilot was conducted to test the discussed procedures, and study the accuracy of self- 
report data. 

Participants 

Participants in this preliminary pilot were students enrolled in a required methods course as well 
as staff in information technologies at a southwestern university (n-1 8), All participants completed a brief 
survey; twelve completed log-recordings. 

Procedures 

Participants completed a brief, three-question survey regarding web-based use, shown in Table 3, 
Table 3, Web use survey 



1) In a week, estimate how many times you use email: 

2) In a week, estimate how many times you use the World Wide Web: 

3) In a week, estimate how many times you use File Transfer Protocol: 



Frequency was described to participants as how many times email was accessed during a day over 
a week period, not as initial access (i.e., opening their email for the first time that day) or how many emails 
they read or sent. For example, if a user had multiple applications open over a period of time during the 
day, email frequency was how many times the user went back to view their email, regardless of read or sent 
email. Accessing the World Wide Web was explained in the same manner. As File Transfer Protocol is 
fairly discrete, frequency was described as how many times the application was accessed during a week. 
Twelve participants completed the log-recording of web-use specific to email, World Wide Web 
and File Transfer Protocol, Participants were asked to keep a frequency log in the morning, afternoon and 
evening, over a 5-7 day period. All participants in this phase completed six-seven days of log-data. For 
data analysis, we used six days of log data from each participant. 



Results 

Correlational analysis indicated that the measures of reporting usage of FTP and email are 
consistent, but Web browsing is not (see Table 4) 

Table 4, Correlation Coefficient between Daily Los and Surrey Data 



Usage 


r 


P 


Email 


,79 


,0039 


Web 


,58 


,0625 


FTP 


,91 


,0001 
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Difference scores between two measures for usage of email, FTP, and Web browsing were 
standardized and plotted in histograms for examining bias and variance. Since bandwidth (number of bins) 
affects the appearance of the distribution, different number of bins and ticks were tried to gain a thorough 
view of the data. It was found that the measurement errors of usage of email, FTP, Web browsing were 
variable errors rather than bias. Figures 1 -3 indicate that although most difference scores were centered 
around zero, the spread went as far as three standard deviations and thus the reliability of self-reported data 
was still questionable. 



Figure L Standardized Difference Scores ofUsase of Email 




Figure 2. Standardized Difference Scores of Usa 2 e of FTP. 
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Figure 3. Standardized Difference Scores of Usase of Web browser. 




Discussion 

Pertaining to reliability estimation, tremendous attention has been paid to internal consistency, 
which is measured by Cronbach Alpha Coefficient. Internal consistency is just one of several aspects of 
reliability. Other aspects such as stability over time and equivalence between different measures are 
considered more important when the accuracy of data is suspect. The approach introduced in this paper 
addresses the issue of stability and equivalence. Since users are measured in different times, stability is 
taken into account of reliability estimation. Because different forms of measurement are employed, 
equivalence is also included as a reliability component. 

Since a demanding commitment (writing a log everyday) is required in this study, the number of 
participants is small and the measurement is as simple as possible (three questions only). -External 
motivation (e.g. extra bonus points) will be provided in subsequent studies so that more subjects will be 
obtained and more questions will be asked. 
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Abstract 

This research presents a survey of doctoral dissertations completed by the students at the 
Educational Communications and Technology Program (ECT) at the University of Wisconsin- 
Madison from 1977 to 1999, The survey results present not only a list of valuable dissertation 
titles and analysis of the research projects, but also demonstrate the evolution of research 
methodologies and theories applied in educational technology research, A special tribute is paid 
to Dr. Ann De Vaney, founder and Director of the ECT Program for her educating and mentoring 
the majority of the scholars in this group and how she has influenced the field. This project is in 
honor of Dr, Ann De Vaney, 



Purpose of the Study 

The successful and unique ECT program created and directed by Dr. Ann De Vaney since 1976 
provides students with opportunities of not only learning how to use technology, but also studying social 
and cultural issues related to technology. As Dr, De Vaney explains that she “wanted a the new program to 
be an alternative to existing ones in educational technology.” {School of Education News, p. 8), The ECT 
program has been nationally unique in offering an “opportunity to pursue a wide range of critical studies on 
the use of educational media and technologies in the curriculum, in the workplace, and in society. These 
critical studies address the social, philosophical, political, and cultural foundations of technology in 
education.” (Graduate Program in ECT). Alumni of the program now work as professors, technology 
coordinators, teachers, and instructional designers, and businesses all over the country, and around the 
world. Most of the alumni from the Ph.D. program are professors or in other research related positions. 

In this research project, I surveyed doctoral dissertations completed by the students at the 
Educational Communications and Technology Program; analyzed the topics, methodologies, and theories 
used in the dissertations, surveyed (in process) the alumni in the Ph.D. program, compiled (in process) a list 
of publications by the alumni, and will conduct a citation analysis of publications by these scholars. 

Methodology 

Three research methodologies were used for the study: survey, content analysis, and citation 
analysis. Survey methodology was used to gather a complete list names of the dissertators and their major 
professors, date of completion, and title of the dissertations the ECT program office. Dissertation abstracts 
were collected from the University of Wisconsin-Madison Library and the International Dissertation 
Abstracts by UMI, Survey methodology was also used to determine where the original dissertators are and 
what their current professional positions and activities are. In order to locate where the alumni are, I 
contacted the University of Wisconsin Alumni Association, verified the subscribers on the AECT-L 
listserv, personally contacted the majors professors of the alumni, and obtained information about them 
from colleagues and friends. Content analysis of the dissertation abstracts was carried out to study the 
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research topics, methodologies, and theories used in their dissertations and see how research methodologies 
and theories in educational technology have evolved throughout the years. I also examined the writings by 
the professors in the ECT program especially works by Dr. De Vaney. The analysis shows that her works 
have influenced tremendously in their dissertation and later scholarly publications. It is my hope that the 
collection of the writings produced by this intellectual group will be helpful and beneficial to many 
scholars in the field. 

Analysis and Results 

A list of valuable dissertation titles with abstracts was complied. The dissertation title list is 
presented later in the article. The analysis was based on the abstracts. The dissertators’ cultural 
backgrounds include American, Arabic, Chinese, Ethiopian, Hispanic, Korean, Puerto Rican, Nigerian, 
Philippine, and South African. These dissertations reflect a variety of most innovative research 
methodologies and theories that are applied to study a variety of issues in educational technology. It 
demonstrates the evolution of research methodologies and theories applied in educational technology 
research since the beginning of the ECT program. If one closely examines Dr. Ann De Vaney’ s works, we 
in educational technology “would do well to consider the following paths she [Ann] has gone down, be 
they her research topics, methodologies, and philosophies; be they the people about whom she is interested; 
or be they the ethical stance she takes.” (Nichols, 2000). Her writings go beyond the realm of empirical and 
quantitative research methodologies to explore the intersections of education, technology and cultural 
issues. The use of technology for democracy, human freedom, dignity, social standing, social/cultural 
issues and meaning in educational technology has been of particular interest, especially the role of 
minorities, women, children, and issues brought about by postmodernism. As Nicolas pointed out that Ann 
“has been concerned with helping and improving the field of educational technology.” (Nicholas, 2000). 
The dissertations completed under her guidance reflect greatly the methodologies, theories, philosophies of 
Dr. De Vaney. A short list of publications by Dr. Ann De Vaney is provided at the end of this article. Based 
on the analysis on the dissertation abstracts, brief summaries of methodologies, theories, and subject topics 
studied by these authors are provided below: 

Various Methodologies Used (It is arranged by dissertation date). 



Experimental Design 

Observation 

Survey 

Interview 

Case Study 

Historical Descriptive Case Study 

Content analysis 

Semiotic Analysis 

Rhetorical and Structural Analysis 

Formalist Method of film analysis 

Post structural textual analysis 

Discourse analysis 

Cultural analysis 

Phenomenological methodology 

Various Methodologies Used 

Poststructural/cultural studies 

Combination of qualitative and quantitative methods 

And other methods. 
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Theories Used (It is arranged by dissertation date). 



Behaviorist learning theory 

Cognitive learning theory 

Constructivism 

Gestalt theory 

Semiology 

Formalism 

Critical pedagogy 

Literary theory 

Social learning theory 

Feminist poststructuralist theories 

Situated learning theory 

Reader-response theory 

Chicana feminist theory 

Social reader theories 

Critical feminism theory 

Critical race theory 

Queer theory 

And other theories 

Subject of the Studies (It is arranged by dissertation date). 

Audio-visual centers 
Photographs 
Educational films 
Motion pictures 
Instructional television series 
Media centers 
Still visuals 

Instructional technology in other countries 

Teleconference 

Television news 

Media education 

Visual and verbal learning 

Instructional television presentation forms 

Television news 

Computer anxiety in teachers 

Instructional computing 

Computers in schools 

Students and videodisc system 

Ethnicity and computer equity 

Meaning studies (dealing with films, art works, hypermedia, television program, and other subject 
areas). 

Interactive video and learning styles 

Discourse analysis 

Women in audiovisual education 

Learner-centered software design and development 

Girls and computers 

Race, ethnicity, gender in educational technology 
Democracy in distance education 
And other topics. 
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Figure /. PhD^s in Educational Communications and Technoloev 
University of Wiscons in-Madison 



Major Professor 


Student 


Date of completion 


De Vancy, Ann 


Lindas, Kathleen, Riley 


Will 


De Vancy, Ann 


Chute, Alan 


?/6/78 


De Vancy, Ann 


Mclsaac, Marina 


8/22/80 


Dc Vancy, Ann 


Strcibcl, Michael J. 


12/22/80 


De Vancy, Ann 


Robinson, Rhonda 


5/17/81 


De Vancy, Ann 


Jones, Maxine 


8/23/81 


De Vancy, Ann 


McAfcc, Dianne 


12/20/81 


De Vancy, Ann 


Englcsby, John 


12/19/82 


Dc Vancy, Ann 


Edison-Swift, Paul 


5/22/83 


De Vancy, Ann 


Slattery, Karen 


snim 


De Vancy, Ann 


Muffolctto, Robert 


5/21/84 


De Vancy, Ann 


Wilson, Bonnie 


5/21/84 


De Vancy, Ann 


Miller, Jacqueline 


8/26/84 


Dc Vancy, Ann 


Madarsha, Kim 


5/21/85 


De Vancy, Ann 


Salcedo, Annunciacion 


8/28/85 


De Vancy, Ann 


Weiss, William 


12/22/85 


De Vancy, Ann 


Kervin, Denise 


12/23/85 


De Vancy, Ann 


Holtz, Brian 


8/28/86 


Dc Vancy, Ann 


Krentz, Roger 


12/23/86 


De Vancy, Ann 


Johnson, Neil 


5I2US1 


De Vancy, Ann 


Knupfer, Nancy 


5/21/87 


De Vancy, Ann 


McLcllan, Hilary 


5/21/87 


De Vancy, Ann 


Mannaa, Ibrahim 


5/21/88 


De Vancy, Ann 


Pomper, Maijoric 


12/23/88 


Dc Vancy, Ann 


Kcnnard, Margot 


milS9 


De Vancy, Ann 


Elencs, Claudia Alcjandra 


5/16/93 


Dc Vancy, Ann 


Kirby, Lcnorc 


5/16/93 


Dc Vancy, Ann 


Ma, Yan 


5/16/93 


De Vaney, Ann 


Shutkin, David 


Slum 


Dc Vancy, Ann 


Butler, Rebecca 


5/21/95 


De Vancy, Ann 


Hollenbeck, John 


si\mi 


Dc Vancy, Ann 


Pettit, Myragcnc 


8/24/97 


Ellsworth, Elizabeth A. 


Erdman, Barbara 


12/23/89 


Ellsworth, Elizabeth A. 


Vazquez, Osvaldo 


8/25/91 


Ellsworth, Elizabeth A. 


Omcr, Marilyn 


S123192 


Ellsworth, Elizabeth A. 


Fuller, Laurie 


12/22/96 


Ellsworth, Elizabeth A. 


Clifton, William 


8/23/98 


Lukowksy, Jeffrey 


Koetting, J. Randall 


7/12/79 


Lukowksy, Jeffrey 


Emekauwa, Emcka 


5/21/83 
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BEST COPY AVAILABLE 



Lukowsky, Jeffrey 


Considine, David 


8/23/81 


Lukowsky, Jeffrey 


Mereba, Tamrat 


8/21/83 


Streibel, Miehael J. 


DeGraff, Jeffrey 


8/28/85 


Streibel, Miehael J. 


Silverstein, Mark 


12/23/87 


Streibel, Miehael J. 


Taylor, Charles 


5/21/88 


Streibel, Miehael J. 


Elefson, Jean 


12/23/89 


Streibel, Miehael J. 


Gamsky, Diane 


5/17/92 


Streibel, Miehael J. 


Hass, Jaequeline 


12/23/92 


Streibel, Miehael J. 


Yeh, Sung-Chuan 


12/23/93 


Streibel, Miehael J. 


Guekenberg, Thomas 


8/27/95 


Streibel, Miehael J. 


Hanor, Joan 


8/27/95 


Streibel, Miehael J. 


Howell, Dusti 


8/27/95 


Zimmerman, Harry 


Andrews, Cheryl 


12/22/79 



List of Dissertations from 1977 to 1999 

7/7/1977 Lindas, Kathleen Riley De Vaney, Ann 

Identification Of Criteria And Evidences For The Evaluation Of The Instructional Materials Center 
Program In A Secondary School 

7/6/1978 Chute, Alan Gilbert De Vaney, Ann 

An Experimental Study Investigating The Comparative Effectiveness Of A Color And A Monochrome 
Version Of 

An Instructional Film Presentation On Incidental And Task-Relevant Discrimination Learning. 

7/12/1979 Koetting, J. Randell Lukowsky, Jeffrey 

Towards A Synthesis Of A Theory Of Knowledge And Human Interests, Educational Technology And 
Emancipatory Education : A Preliminary Theoretical Investigation And Critique 

12/22/1979 Andrews, Cheryl Ann Zimmerman, Harry 

The Audio-Visual Center Phenomenon; An Historical Survey Of The Forces Influencing The Foundation 

And Development Of One Audio-Visual Center 

8/22/1980 Mclsaac, Marina Stock De Vaney, Ann 

The Effects Of Instruction In Black And White Photography On Certain Perceptual And Aesthetic Skills 

Among 4th And 5th Graders 

1 2/22/1 980 Streibel, Michael J. De Vaney, Ann 

The Role Of Stimulus Size On Performance In The Embedded-Figures-Test And In The Rod-And-Frame- 

Test And The Implications Of This Role For The Field-Dependence - Independence Construct 

5/17/1981 Robinson, Rhonda Sue De Vaney, Ann 

Affective Responses To The Literary And Cinematic Elements In An Educational Film: A Descriptive 

Investigation 

8/23/1 98 1 Considine, David Mac Lukowsky, Jeffrey 

The Depiction Of Adolescent Sexuality In Motion Pictures: 1930 - 1980 

8/23/1981 Jones, Maxine Holmes De Vaney, Ann 

Attitudes Toward Four Instructional Television Series: A High School Case Study 
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12/20/1981 McAfee Hopkins, Dianne Talmadge De Vaney, Ann 

A Study To Determine The Presence Of Observable Conditions Of Positive Self-Concept In Elementary 
School Media Centers 

12/19/1982 Englesby, John Hammond De Vaney, Ann 

The Influence Of Still Visuals With Varying Degrees Of Realism On The Learning Of Primary Children 

With High And Low Spatial Ability 

5/21/1983 Emekauwa, Emeka Linus Eze Lukowksy, Jeffrey 

Instructional Technology In Nigerian Education 

5/22/1 983 Edison-Swift, Paul David De Vaney, Ann 

A Study In Teleconference Interaction Analysis 

5/22/1983 Slattery, Karen Louise De Vaney, Ann 

The Implications Of Cognitive Learning Theory For Learning From Television News 

8/21/1983 Mereba, Tamrat Lukowsky, Jeffrey 

Education, Mass Participation And The Impact Of Communication On The Rural Development Campaign 

In Ethiopia: 1974-1978 

5/21/1984 Muffoletto, Robert Nicholas De Vaney, Ann 

Towards A Critical Pedagogy Of Media Education 

5/21/1984 Wilson, Bonnie De Vaney, Ann 

The Relationship Of Field Dependence - Independence And Prior Knowledge Of Passage Content To 

Recognition Of Main Ideas And Details In Illustrated And Nonil lustrated Expository Text 

8/26/1984 Miller, Jacqueline Hilda De Vaney, Ann 

Spatially Cohesive Visuals As Mediators Of Verbal Learning And Recall In Second Language Acquisition 

(Interitem Spatial Organization, Mnemonics) 

5/21/1985 Madarsha, Kim (Kamal B.) De Vaney, Ann 

A Study In The Interpretation Of Symbolic Visuals (Literacy, Semiotics, Malaysia) 

8/28/1985 DeGraff, Jeffrey Thomas Streibel, Michael J. 

Towards An Analysis Of Instructional Television Presentational Forms: A Case Study (Semiology, 
Formalism, Materialism, Representation, Film) 

8/28/1985 Salcedo, Anunciacion Martinez De Vaney, Ann 

A Rhetorical And Structural Analysis Of Instructional Television 

12/22/1985 Weiss, William Walton De Vaney, Ann 

A Visual Analysis Of The Coverage Given The 1984 Presidential Compaign By Cbs Television On The 
CBS Evening News With Dan Rather (Journalism, Semiotics, Elections) 

12/23/1985 Kervin, Denise Jeanne De Vaney, Ann 

Structure And Meaning: A Semiotic Analysis Of Network Television News 

8/28/1 986 Holtz, Brian Davi De Vaney, Ann 

The Implications Of Alternative Placement Of Video Subtitles, Perception Of Program Content By Adults 

With Varying Figure Disembedding Ability 

1 2/23/1 986 Krentz, Roger Franklin De Vaney, Ann 

A Study Of Selected Competencies Of Full-Time School district Media Directors As Perceived By Three 
Groups Of Educators 
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5/21/1987 Johnson, Neil David De Vaney, Ann 

Effects Of Inservice Training On Writing Apprehension And Computer Anxiety In Elementary School 

Teachers 

5/21/1 987 Knupfer, Nancy Nelson De Vaney, Ann 

A Survey Of Teachers' Perceptions, Opinions And Attitudes About Instructional Computing: Implications 
Regarding Equity 

5/21/1987 McLellan, Hilary De Vaney, Ann 

Microcomputers In A High School Astronomy Class: A Case Study 

12/23/1987 Silverstein, Mark Alan Streibel, Michael J. 

Social Interaction Analysis Of Elementary School Students And A Level One Videodisc System In An 
Educational Environment 

5/21/1988 Mannaa, Ibrahim Mohammed De Vaney, Ann 

The Use Of Teletext To Enhance Instructional Television Programs 

5/21/1988 Taylor, Charles Andrew Streibel, Michael J. 

A Comparison Of Black And White Students' Attitudes Towards Factors Influencing Computer Equity In 
Predominantly White Urban High Schools 

12/23/1988 Pomper, Marjorie De Vaney, Ann 

A Semiotic Analysis Of Logo In Practice 

8/27/1989 Kennard, Margot Elizabeth De Vaney, Ann 

The Corporation In The Classroom: The Struggles Over Meanings Of Menstrual Education In Sponsored 

Films, 1947-1983 

12/23/1989 Elefson, Jean O’Brien Streibel, Michael J. 

How Do Agricultural Journalists Write? The Study Of Social-Cognitive Writing Behaviors Of A 
Specialized Writing Process (Journalists, Writing) 

12/23/1989 Erdman, Barbara Elaine Ellsworth, Elizabeth A. 

Lesson Into Film: A Formalist Analysis Of The Teaching Film 

8/25/1991 Vazquez Fernandez, Osvaldo Ellsworth, Elizabeth A. 

An Ideological Analysis Of 'Sesame Street': An Original Study (Educational Television, Puerto Ricans) 

5/17/1992 Gamsky, Diane Streibel, Michael J. 

8/23/1 992 Orner, Marilyn Beth Ellsworth, Elizabeth A. 

Teaching Otherwise: Feminism, Pedagogy And The Politics Of Difference 

12/23/1992 Hass, Jacqueline Streibel, Michael J. 

Educational Computing And The Practice Of Teaching: A Case Study (Computing, Instruction) 

5/16/1993 Elenes, Claudia Alejandra De Vaney, Ann 

Latina Representation In 'Square One Tv' (Educational Television, Television) 

5/16/1993 Kirby, Lenore De Vaney, Ann 

A Reader-Response Analysis Of Hypermedia (Reader Response, Cai) 

5/16/1 993 Ma, Van De Vaney, Ann 

A Reader-Response Analysis of 'A Book From The Sky': A Postmodern Educational Enterprise 
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12/23/1993 Yeh, Sung-Chuan Streibel, Michael J. 

Perceptual Learning Style And Academic Intrinsic Motivation Level: The Effect Of Using Interactive 
Video On Student Post-Treatment Preference And Performance In A High School Chemistry Course 

5/22/1994 Shutkin, David Seth De Vaney, Ann 

The Deployment Of Information Technology And The Child As Subject: A Discourse Analysis Of School 

Restructuring Initiatives, Psychological Research And Educational Policy 

5/21/1995 Butler, Rebecca P. De Vaney, Ann 

Women In Audiovisual Education, 1920-1957: A Discourse Analysis 

8/27/1995 Guckenberg, Thomas Frederick Streibel, Michael J. 

The Design Approach To Software Development Utilizing Children As The Designers Of Learner- 
Centered Logo-Based Mathematics Software 

8/27/1995 Hanor, Joan Helena Streibel, Michael J. 

The Aesthetic Nature Of Young Girls’ Classroom Engagement With Computers 

8/27/1995 Howell, Dusti D. Streibel, Michael J. 

Indicators Of The Emancipatory Interest Component Of Teaching (Ira Shor) 

12/22/1996 Fuller, Laurie S. Ellsworth, Elizabeth A. 

Disrupting Whiteness: Race, Queemess And Pedagogy (Queer Theory, Gender) 

5/18/1997 Hollenbeck, John Alan De Vaney, Ann 

Democracy In Computer-Mediated Communications (Distance Education, Computer Conferencing) 

8/24/1997 Pettit, Myragene Elizabeth De Vaney, Ann 

Ameritech's 'Super School’ Project In Three Selected Wisconsin Schools (Telecommunications) 

8/23/1998 Clifton, William Harris Ellsworth, Elizabeth A, 

The 1993 Wisconsin-Michigan Football Game 'Critical Incident’: The Struggles Over Meanings OfUW 
Students By WU Administrators (University of Wisconsin, University Of Michigan, College Students, 
Crisis, Meaning-Making, Civil Disorder) 

?/?/! 999 Price, Todd De Vaney, Ann 

(Title and abstract not available) 



Writings and Scholarship, Service, and Leadership of the Alumni from the Ph.D. 
Program 

A large number of publications by the above scholars has been collected and compiled. Wilson 
databases, ERIC, Art Abstracts/Index, Applied Science and Technology, Humanities Abstracts/Index, 
Social Science Abstracts/Index, Library Literature, and Inter Library Loan were used to find the 
publications. Additional publication titles are being obtained by surveying each individual author. Many of 
them have played important roles in leading research areas in educational technology, in teaching of 
educational technology, and in serving on national and international organizations. Their publications, their 
current positions, their professional service and contribution to the filed will be presented in a separate 
publication. 
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Learners and the Learning Environment; 



Impact of Technology Use in K-12 Schools in Iowa 

Nancy J. Maushak 
Kuo-Tsai Chen 
Texas Tech University 

The purpose of this study was to examine the impact of the IDEA Star Schools Project activities 
on the learner and the learning environment. A questionnaire based on previous work at NCREL 
and Milken was used to collect data from current and past winners of an exemplary technology use 
award. Results from this study provided valuable information for stakeholders on the change that 
occurs in learners and the learning environment when technology was used effectively. 



Introduction 

Evaluation has always been and continues to be a major component of the Iowa Star Schools 
Project. Results from evaluation activities conducted in 1998 indicated that technology was being 
integrated into the curriculum in Iowa’s schools. Teachers and students were using technology, at least at a 
very basic level. In addition, teachers were beginning to recognize technology as a tool that “allowed them 
to vary the instructional approach and to provide educational experiences that would be difficult to do 
otherwise.” 

Evaluation in 1 999 attempted to take a closer look at the integration of technology and focused on 
the impact of technology on the learner and the learning environment. 



Purpose of Study 

The purpose of this study was to examine the impact of the Iowa Star Schools Project activities on 
the learner and the learning environment. It is important to recognize that project activities have not been 
implemented in isolation, but have been carried out in conjunction with other technology initiatives at the 
local, state, and national level. This limits the interpretation of the data collected and prevents us from 
making a causal link. 

Both models discussed below guided the data collection during the 98-99 Project evaluation 
activities. This study, only one piece of the entire evaluation, served as a pilot for future data collection. 
Changes will be made to the 99-2000 evaluation plan based on the results of this study. Instrumentation 
will be revised based on the emerging model supported by the results of 98-99 evaluation data analysis. 

The work of both the North Central Regional Educational Laboratory (NCREL) and the Milken 
Exchange on Education Technology provided guidance as well as a framework for evaluation and research 
activities. NCREL’s engaged learning model builds on the work of Barbara Means at SRI International. 
Means identified variables indicating the occurrence of effective teaching and learning. Researchers at 
NCREL reviewed other related research to expand this list and developed a framework of 26 indicators of 
engaged learning (Table 1). 

The NCREL engaged learning model provides a starting point to look at the impact of technology 
on teaching and learning. The Educational Technology Seven Dimensions of Progress Framework 
developed by Milken Exchange on Education Technology provided a complementary model expanding the 
guidance for research in this area (Lemke & Coughlin, 1998). Table 2 describes the seven dimensions and 
the indicators provided for each dimension. 



Table 1 Indicators of Ensased Learning 

Variable Indicator of Engaged Learning 
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Vision of Learning 

Tasks 

Assessment 

Instructional Model 
Learning Context 

Grouping 

Teacher Roles 

Student Roles 



Responsible for learning 
Strategic 

Energized by learning 

Collaborative 

Authentic 

Challenging 

Multidisciplinary 

Performance-based 

Generative 

Seamless and ongoing 

Equitable 

Interactive 

Generative 

Collaborative 

Knowledge-building 

Empathetic 

Heterogeneous 

Equitable 

Flexible 

Facilitator 

Guide 

Co-leamer/co-investi gator 
Explorer 

Cognitive apprentice 

Teacher 

Producer 
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Table 2 Seven Dimensions of Prosress 



Dimension 


Indicators 


Learners 


Fluency 

Strengthening the basics 
Developing higher level skills 
Increased relevancy 
Motivation to learn 
Recognition of tradeoffs 


Learning Environments 


Learning context 
Learning content 
School culture 
Technology access 
Information and communication 


Professional Capacity 


Core technology fluency 
Curriculum, learning and assessment 
Professional practice and collegiality 
Classroom and instructional management 


System Capacity 


Vision 
Leadership 
Ensuring capacity 
Systems Thinking 


Community Connections 


Commitment 

Collaboration 

Clarity 

Communication 


Technology Capacity 


Installed Base 
Connectivity 
Technical support 
Client orientation 
Facilities 

Deliverables and benchmarks 
Data collection/interim progress 


Accountability 


Data-driven decision making 
Communication 



The purpose of this study was to serve as a pilot study to examine the impact of technology 
integration on learners and the learning environment, the first two dimensions of Milken* s Dimensions of 
Progress and to provide important formative evaluation data for decision making. 



Instrument Development 

The instrument used in this study was adapted from a questionnaire used in a technology study in 
Virginia conducted by the Milken Exchange on Education Technology, NCREL, and SRI International 
(Report to the Commonwealth of Virginia, 1998). The original instrument addressed all seven dimensions 
for gauging progress in technology in schools. The instrument used in this study focused on only the 
dimensions of Learners and Learning Environment. The indicators included in these two dimensions and 
influenced by NCREL*s engaged learning model guided revisions. 

The final instrument included four sections summarized in Table 3. As indicated, questions were 
grouped by the indicators associated with each of the two dimensions under study. The constructs from the 
learner dimension formed the basis of analysis of this study. 
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Table 3: Summary Survey Instrument 



Section 


Constructs 


Number of Questions 


Response Set 


Section 1 


Demographics 


8 


Check box 


Section 2 


Fluency 


4 


5 point Likert Scale 


Learners 


Basics 


4 






Higher Order Thinking 


4 






Relevancy 


4 






Motivation 


7 




Section 3 


Student engagement with 


17 


5 point Likert Scale 


Learning 


technology 






Environment 


Teacher engagement 


14 






with technology 








Teacher change 


13 






Support 


9 


Check box 




Barriers 


18 


5 point Likert Scale 


Section 4 


Impact 




Open Response 



Reliability Analysis 

As indicated by Suen and Stevens (1993), the reliability of the subscales needsto be estimated 
when using subscale scores or constructs as the unit of analysis. Cronbach alpha is designed to estimate the 
reliability of a composite score from the responses to a number of items. 

Cronbach alpha coefficients were determined for each of the five learner constructs. Table 4 
shows the Cronbach alpha reliability estimate for each construct. These estimates indicated that the 
constructs were reliable, with coefficients ranging from .73 to 83. The reliability estimate determined for 
the overall Learner dimension was .94 indicating the reliability of this section. 



Table 4: Reliability Estimates of Learner Constructs 



Construct 


Reliability Estimate 
(Cronbach alpha) 


Fluency 


.73 


Basics 


.83 


Higher Order Thinking 


.80 


Relevancy 


.78 


Motivation 


.83 


Cronbach alpha coefficients were 


also determined for three of the learning environment 


constructs. These are presented in Table 5. 
coefficients ranging from .82 to 90. 


These estimates indicate that the constructs are reliable, with 


Table 5: Reliabilitv Estimates ofLearnin 2 Environment Constructs 


Construct 


Reliability Estimate 
(Cronbach alpha) 


Student engagement 


.82 


Teacher engagement 


.86 


Teacher change 


.90 
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Sample Population 



Participants in this study were current and past winners of the Showcase on Technology award, 
part of the Iowa Star Schools Project. This award has been given out annually for the past four years. It 
recognizes K- 1 2 and higher education teachers who are using technology in innovative ways. A summary 
of these projects can be found on Iowa’s Star Schools Project web site ( www.iptv.org/iowa_databascA . 

Showcase project leaders were contacted and provided a list of all past winners. Only K-12 
teachers were included in this study. In spring 1999, 71 surveys were sent out and on first analysis 43 
surveys were returned for a response rate of 61%. However, on closer examination, four responses were 
found to be unusable (one envelope returned empty, and three unmarked surveys returned) lowering the 
response rate to 55%. Though this is lower than desired, due to the time of year and the fact that this was a 
pilot study, it was felt that the response rate was sufficient. 

Representatives from all grade levels, pre-K to 12*^ and ungraded, responded. Respondents were: 

• predominately middle school (44%) and high school (31%) teachers (Table 6); 

• represented all content areas with the highest response from English/Language 
Arts/Reading (51%) and Science (46%) (Table 7); 

• mostly female (67%). 



Table 6: Teachins level Table 7: Content Area 



Level 


N 


% 


Subject Area 


N 


% 


High School 


12 


30.8 


Computers/technology 


5 


12.8 


Middle School 


17 


43.6 


English/Language Arts 


20 


51.3 


Elementary 


9 


23.1 


Foreign Language 


1 


2.6 


Multi-Level 


1 


2.6 


History/Social Sciences 


10 


25.6 








Mathematics 


11 


28.2 








Science 


18 


46.2 








Vocational Education 


3 


7.7 








Other 


3 


7.7 



The majority (74%) indicated that they had access to a computer with modem at home; 67% used 
their computer at home for Internet access once or more a week or daily (Table 8). Internet access from 
school was higher with 80% indicating the used the Internet daily. 



Table 8: Internet Use 



Internet Use 


Home 

N 


% 


School 

N 


% 


Daily 


19 


48.7 


31 


79.5 


Once or more a week 


7 


17.9 


8 


20.5 


Once or more a month 


1 


2.6 


0 


0 


Less than monthly 


0 


0 


0 


0 


Never 


12 


30.8 


0 


0 



Findings 

Data collected from Showcase winners were analyzed in four ways. First the frequencies were 
calculated. Second, the overall mean scores for the constructs allowed some determination of the impact of 
technology on the learner and teacher, and provided some indications of the overall use of technology in 
the classroom. Third, analysis of barriers and incentives provided guidance for future training; and last, 
analysis of open-ended responses provided further information of the impact of technology and indicators 
of change. 
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Learner Dimension: Individual Items 

In general, it appeared that teachers feel that technology impacts students on a variety of levels. In 
assessing weak areas, a level of 25 percent indicating no change, negative change, or not sure was set by 
the researcher; items where one fourth or more of the teachers responded negatively were identified as 
areas of concern. For 1 6 of the 23 items, more than three-quarters of the teachers indicated a positive 
change. More than 75 percent indicated technology had impacted students through improvement in the 
following areas: 

• acquisition of new technology skills (100%); 

• competency in technology skills (97%); 

• number of assignments produced with technology (97%); 

• engagement in activities beyond “rote*’ learning (97%); 

• engagement in inquiry-based learning activities (95%); 

• use of technology in a variety of educational settings (92%); 

• “real life” applications of subject (92%); 

• ability to communicate results in a variety of ways (90%); 

• breadth of students’ understanding of the subject (87%); 

• independence as learners (87%); 

• attitude toward subject (87%); 

• ability to construct knowledge and solve problems (85%); 

• depth of students’ understanding of the subject (82%); 

• collaboration with others in school (80%); 

• overall interest in school (77%); and 

• attentiveness/engagement in class (77%); 

There were seven areas where technology had no influence or a negative influence on learners. 
Areas where more than 25% of respondents indicated no change, negative change or not sure included: 

• school attendance on days when technology is scheduled to be used (69%); 

• school attendance in general (67%); 

• development of work ethic (4 1 %); 

• understanding of the “basics” (28%); 

• collaboration with others outside school (28%); 

• attempts to go beyond minimal assignment (29%); and 

• overall academic achievement (26%). 



Learning Environment: Individual Items 

Student Engagement 

Results of student engagement in the learning environment dimension indicated students were 
using a limited variety of technologies. Analysis of items related to student use of technology identified 
only two areas where three-fourths or more of respondents indicated technology being used by students at 
least 30 minutes a week. These two areas were: 

• use of computers for any educational purpose (95%); and 

• use of word processing software (85%). 

Over 50% indicated that students were using the two areas related to research at least 30 minutes a 
week. These two areas were: 

• research using Internet (64%) : and 

• research using CD-ROM (62%). 

Technologies where fewer than 25% of the respondents indicated students were using them at least 
30 minutes a week included: 
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• spreadsheet software ( 1 0%); 

• videodiscs (15%); 

• database software ( 1 8%); and 

• create/maintain WebPages (18%), 



Teacher Engagement 

Analysis of items related to teacher use of technology identified three items where at least 75% of 
the respondents indicated they were using the technology a minimum of 30 minutes a week. These items 
included; 

• use of computers for any work-related purpose (87.2%); 

• use of word-processing software to create/update course materials (82.1%); and 

• use of email to communicate with colleagues inside and outside the school 

(82.0%); 

Results indicated that teachers are not using email to communicate with parents. In addition, there 
were four other technologies where less than 25% of the respondent indicated that they used the technology 
a minimum of 30 minutes per week included: 

• Research using CD-ROM (23 . 1 %); 

• Use Internet to provide information to the community about your 

classroom/school. (20.6%); 

• Create/manage/analyze spreadsheets ( 1 5.4%); 

• Participate in an interactive video environment (e.g. ICN session) for 
professional development (15.4%); and 

• Use email to communicate with parents (5.2%). 

Teacher Change 

It appears that not only does technology impact learners it also influences teachers (Table 16). 
Similar to the analysis of learner dimension items, a level of 25% indicating no/negative change was set by 
the researcher to identify items of concern. On seven items, three-fourths or more indicated a positive 
influence of technology on their teaching behaviors. These areas were: 

• share ideas and skills with colleagues (92%); 

• overall quality of instruction being delivered (90%); 

• personal repertoire of instructional strategies (90%); 

• amount of materials and resources being used in classes (90%); 

• number of changes being made in the curriculum (85%); 

• personal participation in instructional planning at either the department or school 
level (80%); and 

• efficiency and/or effectiveness of classroom management (80%). 

There were six areas where teachers indicated there was no/negative change or they were not sure. 
These areas included: 

• relationships with parents and communities (5 1 %); 

• relationships with students (38%); 

• personal participation in instructional planning at district/AEA or state level 
(36%); 

• sense of empowerment to address school issues (31%); 

• collaboration with colleagues to identify goals, make decisions, and solve 
problems (28%); and 

• general morale (26%). 
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Constructs 

The five-point Likert scale items related to the Learner dimension were grouped into five 
constructs and mean scores were calculated. As shown in Table 9, teachers appeared to feel that the 
integration of technology changed students in a positive way in all five areas. Not unexpectedly, the 
greatest improvement was in the area of fluency in the use of technology (mean=4,55). 



Table 9: Learner Dimension Construct Means 



Construct 


Mean 


SD 


Fluency 


4,55 


,35 


Basics 


4,14 


.54 


Higher Order Thinking 


4,40 


,49 


Relevancy 


4,26 


.44 


Motivation 


4,04 


.52 



Means for three areas in the Learning Environment dimension were also calculated. The 
engagement constructs used the same scale and are shown below. The scale used for engagement constructs 
was in increments of time where l=none, 2=less than 15 minutes, 3= 15 to 30 minutes, 4=30 to 60 minutes, 
and 5=more than 60 minutes. Teachers appeared to be using the technology more than students. Both 
groups were using technologies on average between 15 and 30 minutes a week. 



Table JO: Learning Environment Encasement Construct Means 



Construct 


Mean 


SD 


Student engagement 


2.40 


,59 


Teacher engagement 


2,98 


,64 



Teacher change used a different scale where 1 equaled much declined to 5 equaled much 
improved. The construct mean for teacher change was 4,19, This indicated that teachers feel that their 
personal use of technology somewhat improved what they do and how they do things in the classroom, in 
communication and in curriculum preparation. 



Incentives and Barriers 

Respondents were asked to identity the ways they were encouraged to use technology, A full 
summary of responses is shown in Table 1 1, The most common incentives were use of school technology 
in evenings/weekends/school holidays/or summer months (77%); and technology resources for media 
center and/or classroom (74%), Few teachers indicated that they had received salary incentives for the 
effective use of technology (3%), 

Table 11: Percent of respondents identifying incentives received for usins technolosv 



Incentives % (n=39) 

Release time for planning 25.6 

Schedule changes for teachers that allow collaborative learning and planning 25,6 

Technology resources for media center and/or classroom 74.4 

Use of school technology in evenings/weekends/school holidays/or summer months 76.9 

T echnology certification 1 0.3 

Expectation/requirement that faculty use technology as learning tool 5 1 ,3 

Acknowledgement/recognition of effective teacher use of technology 48,7 

Salary incentives for teachers seeking training in technology 17,9 

Salary incentives for teachers effectively using technology as a learning tool 2,6 



Respondents were also asked to rate barriers to the increased use of educational technologies 
(Table 12), Over 50% indicated that time for training (69%) and lack of time for planning and 
implementing (64%) were significant barriers. Student's current level of technology skills and community 
support did not appear to be significant barriers. 
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Table 12: Barriers to Increased use of Educational Technolosies, Frequencies by Percent 

/= not a barrier, 2~ minor barrier, 3= moderate barrier, 4~ significant barrier 





1 


2 


3 


4 


NS 


N 


SD 


Mean 


Cost of purchasing computers and 
other hardware 


5.1 


12,8 


35,9 


43.6 


2.6 


39 


,87 


3,21 


Cost of making structural changes 
to accommodate technology 


12,8 


12,8 


35,9 


35,9 


2.6 


39 


1.03 


2,97 


Cost of technical support and 
training 


7.7 


30,8 


33,3 


28,2 


0 


39 


,94 


2,82 


Cost of maintenance and repair 


5.1 


28.2 


38,5 


25,6 


2,6 


39 


.88 


2,87 


Cost of computer software and 
upgrades 


5.1 


17.9 


35,9 


41,0 


0 


39 


,89 


3,13 


Availability of good models to 
follow in integrating technology 


17.9 


30.8 


25.6 


20.5 


5,1 


39 


1,04 


2,51 


Availability of resources that 
integrate technology (e,g, 
textbooks) 


12.8 


25.6 


43,6 


12.8 


5.1 


39 


,90 


2,59 


Availability of software that 
matches instructional needs 


20.5 


30.8 


38,5 


10,3 


0 


39 


,94 


2,68 


Time for training 


5.1 


7.7 


15,4 


69.2 


2,6 


39 


.86 


3,53 


Lack of time for planning and 
implementing technology 


2.6 


5,1 


25,6 


64.1 


2,6 


39 


,72 


3,55 


Teachers’ current level of 
technology skills 


10.3 


30,8 


30.8 


28.2 


0 


39 


,99 


2,77 


Students’ current level of 
technology skills 


46.2 


30,8 


20,5 


2.6 


0 


39 


,86 


1.79 


Teacher interest/support 


17.9 


33,3 


35,9 


12.8 


0 


39 


,94 


2,44 


Administrative support 


25,6 


33,3 


25,6 


15,4 


0 


39 


1,03 


2,31 


Community support 


38,5 


25,6 


23.1 


2,6 


0 


39 


.90 


1,89 


Number of computers available for 
teachers 


25.6 


28,2 


20.5 


25,6 


0 


39 


1.14 


2,46 


Number of computers available for 
students 


20.5 


30.8 


20,5 


25,6 


2,6 


39 


1,11 


2,53 



Open-ended Reponses 

The constant-comparative method was used to examine the responses to the open ended questions. 
Responses to noticeable change in academic achievement were categorized into six areas. These categories 
in order of frequency were fluency, motivation, learning environment, higher order thinking skills, 
relevancy, and basic skills. Fluency in the use of technology was the most frequent response. An 
improvement in basic skills was mentioned the least. Several indicated that no measurable change in 
achievement was apparent. Sample responses for each of the categories are presented below: 

Fluency 

• “student level of comfort using technology has risen tremendously” 

• “more computers and awareness of how to use by students and teachers” 

• “students can use word processor at an earlier age” 

• “students are able to use technology comfortably” 

• “skills towards using the technology have improved” 
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Motivation 



• “student’s interest in and understanding of topics” 

• “student interest in research projects” 

• “students interest in subject matter and motivation to learn new skills” 

• “I think having access to so much information has made students and teachers 
much more eager to learn and explore. This can make for better learners” 

Learning Environment 

• “Our research resources have really mushroomed. Now, when students have a 
question they go to the computer and find out.” 

• “Students are using a wider variety of sources, more likely to look up 
information, more likely to revise information and presentations.” 

• “We now have a computer in every classroom.” 

• “increased access to current information” 

• “we have expanded our curriculum and course offerings” 

Higher Order Thinking Skills 

• “critical thinking skills have improved” 

• “higher order thinking skills have been further developed and enhanced” 

• “the use of technology has made students better independent achievers. 

Relevancy 

• “more use of real world situations” 

• “greater awareness of the world outside our rural community 

Basic Skills 

• “My upper level science classes have shown a clearer understanding of concepts 
through the use of technology” 

• “some improvement in math and reading” 

While a few respondents indicated a rise in assessment or test scores, most identified other sources 
of evidence of academic change or improvement. Repeatedly, teachers identified an improved quality of 
projects and presentations as evidence of academic achievement. They also indicated that communication 
skills had improved based on their observations of students in the classroom. The following quote probably 
says it best: 

“Student projects are most often interesting and well done. Creative thinking and problem 
solving are used to produce quality work using technology. Student portfolios show evidence of 
works created using technology. Students are excited to learn with new technology. Students are 
motivated to use software to broader understanding.” 

Discussion 

Results from this study indicated that technology integration does have an impact on both teachers 
and students. The learning environment is adapting to allow for greater access to the wealth of information 
that is currently available and the even greater explosion of infomiation in the future. This requires changes 
in both the role of the teacher and the student. 

No longer are teachers the only source of information in the classroom. Because of this, students 
need to become adept in accessing and perhaps more importantly evaluating information from a variety of 
sources. They increasingly need to have skills in communication with technology. Already, they are sharing 
their work via email, powerpoint presentations and web pages. These skills closely mirror those needed in 
the business arena. 

Teachers are slowly recognizing that their role in the classroom is changing from leader to 
facilitator. This is not always an easy shift to make and not always is there the support needed for this 
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change to occur. Teachers indicated that student achievement is greater when using technology in the 
curriculum. However, they caution that this achievement is not measured by the usual standardized test. 

This change is more frequently shown in the quality of projects and presentations and in the use of higher 
order thinking skills. It would seem that these are two areas where we would want students to excel. 

The results of this study clearly identify the need for different measures. The types of changes that 
occur in student learning are not reflected in the numeric scores that have been accepted in the past. 
Alternative ways of student assessment need to be developed and those already available need to be 
accepted as indicative of achievement. 

In addition to new measures, there is a need for innovative/quality models that take technology use 
beyond simply doing what “we have always done” in the classroom. We need to move into the realm of 
learning with technology and allowing the technology to become tool to help us manage, organize, 
manipulate, hypothesize, analyze, and synthesize information. In this way, technology moves from merely 
making us more efficient to allowing us to think and process in ways not possible before. 
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Philosophical Inquiry in Instructional Technology: 
The Forgotten Pathway to Learning 
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The vista of instructional technology appears to be changing. While the audio-visual field has 
“grown-up” and become a profession, the field of instructional technology is still maturing. Although no 
longer what it once was, instructional technology appears to be in a veritable state of transition, finding a 
new voice in the information age. The evolution of computer technology continues to influence the way 
we live and learn in the 2V' Century. With research flourishing across disciplines, the future holds promise 
for discovering uncharted territories in human learning. And, a body of knowledge, now unique unto itself, 
is constantly expanding through instructional technology research, as well. Although Instructional 
Technology has always had a strong orientation toward practice (Richey, 1997), the findings of early 
researchers have often exerted influence over new directions in the field. Edgar Dale’s (1946/1996) “cone 
of experience” predates virtual reality and the contemporary constructivist movement; yet, his proposition 
that reality is the basis of all effective learning supports these movements that value authentic learning 
environments. Sometimes, the voices of our founders can help us make sense out of contemporary issues. 
James D. Finn (1953/1 996) felt that our body of systematic theory needed to be constantly expanded by 
research and thinking. It appears as if instructional technology research is thriving - confirming our 
theories, generating new hypotheses - but, the status of thinking within our field could be one of the most 
important, forgotten pathways to learning. 

Philosophy is all about thinking (Morris, 1 999). Occasionally, as researchers and practitioners, we 
become victims of our own routine, replicating steps, over and over again, rarely pausing to think about 
what it all means or to reflect on our values and beliefs as instructional technologists. Philosophy was 
originally a way of life (Marinoff, 1999) and a requirement for living (Morris, 1999); not the abstract, 
academic discipline as it is commonly known today. Back in the days when Socrates proclaimed that the 
unexamined life is not worth living,” philosophy was intended for ordinary people and it was concerned 
with real life and how to live it (Marinoff, 1999). Although the world’s great wisdom tradition has evolved 
into a multiplicity of discourse communities, the central concern of philosophy from ancient time - how to 
think critically - has been relatively absent from the public agenda in recent years (Marinoff, 1999). 
Philosophical inquiry - the importance and uses of critical thinking - remains misunderstood. At its most 
basic level, philosophical inquiry can provide methods for examining the things that we so often take for 
granted during our daily routines. This paper explores philosophical inquiry in the field of instructional 
technology. First, philosophy and the field of Instructional Technology will be examined. Second, four 
dimensions of philosophical inquiry in instructional technology will be explored. The purpose of this 
paper, therefore, is to provide a framework for examining fundamental issues in the field and position 
philosophy as a legitimate method of inquiry in instructional technology. 

Philosophy and the Field of Instructional Technology 

Donald P. Ely could easily be recognized as the leading philosophical thinker in Instructional 
Technology today. His concern about the definition of the field spans 40 years and his early thoughts on 
philosophy and instructional technology were decades ahead of his time. Thirty years ago, when the field 
was not yet a discipline (Ely, 1970), Ely recognized the importance of finding a “. . . utilitarian and 
commonplace usage” (p. 81) of the word “philosophy.” Ely (1970) posited that “It is only right that there 
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should be a philosophy of instructional technology and that it should vary from individual to individual” (p. 
81). Accordingly, philosophy may be interpreted as a composite statement of beliefs and values from 
which personal purpose and direction are derived (Ely, 1970). 

Instructional Technology is a confluence of many disciplines including education, 
communications, the arts and sciences. With such diverse roots, it is easy to understand the differing 
orientations that have emerged in the field, such as behaviorism, cognitivism and constructivism. Yet, 
fields of study such as Instructional Technology do not have philosophies; people do (Ely, 1 970; Smith & 
Ragan, 1999). Therefore, philosophy influences the theories and research that instructional technologists 
deem most valuable (Richey, 1 998; Smith & Ragan, 1999). Personal beliefs and values define what’s 
important to people and they guide and direct behavior. An instructional technologist’s philosophical 
orientation also serves as a device that filters instructional development decisions (Luiz, 1982). Luiz 
(1982) used philosophical inquiry as a framework for exploring designer decision-making and he asserted 
that 

Instructional developers need to know the implications of their decisions when 
advocating one philosophy, rather than another. It is assumed that their personal 
philosophies, implicit in their actions, act as a screening device through which their 
individual decisions are filtered (p. 1 10) 

Therefore, the more we know about our beliefs and values, the more reflective we become in our 
work as instructional technologists. 

Philosophy is also important in our field because it is a foundation for theory (Koetting, 1996; 
Smith & Ragan, 1999; Snelbecker, 1974). The roots of any science can be traced back to philosophical 
origins (Koetting, 1996; Luiz, 1982) and, since the field of instructional technology is built upon solid, 
scientific foundations, there are important implications. 

While such searching for philosophical roots can take on the nature of a rather pointless 
academic game, and while it is often the case that such procedures are used to legitimize 
rather poorly thought-out ideas, it is, nevertheless, true that it is very often difficult to 
understand why a particular scientific theory was formulated without understanding its 
philosophical origins (Snelbecker, 1974, p. 46). 

The prescriptive function of theory may be drawn from various philosophical orientations or 
“filtered” through the personal philosophies of people. Accordingly, theory can be an expression of belief 
(Koetting, 1996; Macdonald, 1995). Since instructional technologists operate from theoretical frameworks 
that are intimately tied to their values (Koetting, 1996; Richey, 1998), it follows that theory and philosophy 
are intimately connected, thus, exerting influence on the field. Philosophical inquiry, therefore, could serve 
to explicate connections between theory and philosophy while providing insight into the choices made by 
instructional technologists. 

Thus far, this paper has suggested that philosophy is important to instructional technologists for 
two reasons: 1) people have philosophies that influence practice; and, 2) theory is derived from philosophy. 
The ultimate goal of philosophy, however, is wisdom (Morris, 1999), which provides depth and usefulness 
in practical matters. For the instructional technologist, the pursuit of wisdom is the connection between 
theory and practice, and philosophy cultivates unique skills and methods of thinking. There are three types 
of skills that are developed through philosophical inquiry: 1) analysis, 2) assessment, and 3) argument 
(Morris, 1999). As a method of thinking, philosophy cultivates the ability to analyze complex problems, 
assess competing claims and prepare arguments, which are a reasoned presentation of ideas (Morris, 1 999). 
While these skills are not necessarily specific to the field of instructional technology, in large part, they are 
aligned with and support the 1998 instructional design competencies and performance statements published 
by the International Board of Standards for Training, Performance and Instruction (ibstpi, 1 999). Further, 
the skill set of philosophy relies upon the use of reason, which is “. . . the power of moving logically from 
one idea to another, of seeing connections of logic or cause and effect, and of inferring conclusions from 
given premises” (Morris, 1999, p. 31). In a field where instructional technologists are continuously thrust 
into choice-making situations, the skills of philosophy appear to be legitimate methods of inquiry in the 
field. 

Philosophical Foundations of Instructional Technology 
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Given the value that philosophy offers instructional technology, the historical foundations for 
future developments in philosophical inquiry will now be explored. The question is where to begin? 

Ertmer and Newby (Ertmer & Newby, 1993) presented two opposing positions on the origin of knowledge 
- empiricism and rationalism - illustrating clear connections between current learning theories and their 
historical foundations in philosophy. Empiricism is based upon the belief that knowledge is derived from 
sensory experience gained through interactions with the environment (Ertmer & Newby, 1993; Morris, 
1946; Schunk, 1991). Rationalism claims that knowledge originates in the mind through reason (Ertmer & 
Newby, 1993; Morris, 1946; Schunk, 1991). Clearly, these positions on the origin of knowledge can be 
linked to behaviorism, cognitivism and constructivism. However, Ely’s (1970) desire to find a sense of 
usefulness in philosophy doesn’t require this type of probing historical analysis. Ely’s approach is more 
practical and begins with the twentieth century since instructional technology is a twentieth century 
movement. Dewey’s ideas about the relationships between experience, learning and theory could be a 
reasonable starting point for studying philosophy in instructional technology because he spurred a variety 
of research and development in education that ultimately influenced the field. However, instructional 
technology emerged from the audio-visual communications movement (Saettler, 1990), and it makes better 
sense to select audio-visual communications as an arbitrary starting point for studying philosophy in 
instructional technology. 

During the late 1920s and 1930s, film utilization practices and the role of visual aids in education 
were “hot” topics. Among the most notable researchers were Knowlton and Tilton (1929) who explored 
the utilization of motion pictures in instruction and Hoban, Hoban and Zisman (1937) who were interested 
in the inherent value of visual aids over verbalism in education. Another prominent educational researcher, 
W. W. Charters, was also studying permanent learning in the 1 930s, and he examined the connections 
between education and the media. Charters was among the first people to use the term “educational 
engineering” (Saettler, 1 990) and his work laid the foundation for the modem, systems approach to 
instruction (Ely, 1970; Seels & Richey, 1994). Although historically rooted in audio-visual 
communications. Charters’ contributions to instructional technology and his influence on the field are far- 
reaching. In fact, it was a student of W.W. Charters, Edgar Dale, whose “cone of experience” became the 
most influential philosophical concept in the field (Ely, 1970). 

Dale’s (1946/1996) “Cone of Experience” is a conceptual model of learning experiences based 
upon a concrete to abstract continuum. The cone also served to synthesize the progressive theories of John 
Dewey, current thinking about audiovisual communications; and, contemporary thought from the field of 
psychology (Seels & Richey, 1994). More importantly, Dale’s contribution was the first attempt at 
integrating learning theory and audio-visual communications (Dale, 1946/1996; Seels & Richey, 1994). 

In the late 1930s, Dale collaborated with Charles F. Hoban, Jr. on several projects concerned with 
the use of motion pictures in teaching. Although Hoban* s research interests and contributions to the field 
explored the relationships among visual aids and the process of learning, his 1956 keynote address at the 
Lake Okoboji Conference was instrumental in moving the field toward a systems orientation (Ely, 1970). 
The application of systems theory in instructional technology was advanced by James D. Finn, who was a 
student of Edgar Dale and stwcd under Hoban in the US Army during World War II. Finn’s vision for 
integrated systems and processes was a compendium of thought that surrounded the emerging field, which 
incorporated the voices of the early founders while blazing a new trail that would ultimately become known 
as instructional technology. 
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While “...the philosophical lineage of Charters-Dale-Hoban-Finn ... yielded the most productive 
thinking about the field ...” (Ely, 1970, p.85), any discussion about the role of philosophy in instructional 
technology truly begins with Jim Finn. McBeath (1972) reminds us that “for Finn, philosophizing is an 
essential component of future planning if we are to go beyond the expedient” (p. ix). 

A Brief History of Philosophy in Instructional Technology 

When Finn (1962/1996), delivered “A Walk on the Altered Side” before a meeting of the John 
Dewey Society in 1962, he prepared for this paper by studying several recent books on educational 
philosophy. Early in his presentation, he defined technology “.. . as a way of thinking about certain classes 
of problems and their solutions” (p. 48), which he felt was a legitimate concern for the educational 
philosopher. At the time, instructional technology was misunderstood, and Finn framed his discussion as an 
indictment of the many educational philosophers who “failed to understand” that technology is a way of 
thinking (Finn, 1962/1996). For Finn (1962/1996), clarification is one of the jobs of the philosopher, and 
since the path was not yet clear, he concluded his presentation by proclaiming that “. . . the vista of 
educational philosophy is more exciting than ever” (p. 54). With respect to the role of philosophy in 
instructional technology, Finn’s (1962/1996) interpretation of technology as, “... fundamentally, a way of 
thinking” (p. 53) is both trenchant and meaningful. 

Ely’s (1970), paper, “Toward a Philosophy of Instructional Technology,” appears to be the first 
exploration of philosophy within the Instructional Technology knowledge base. Ely’s notion of ‘truth’ 
allows us to interpret philosophy as a subjective “filter” that mediates behavior and decision-making 
processes. Ely’s influence appears evident in the 1972 definition of the field (AECT, 1972), which focused 
on “the facilitation of human learning.” The 1972 definition stated that “the uniqueness of educational 
technology, and, therefore, its reason for being, lies in the philosophical and practical approach it takes 
toward fulfilling this purpose” (p. 37). At the time, Dr. Ely was chairman of the Definition and 
Terminology Committee for the Association for Educational Communications and Technology (AECT); 
and, a group of experts and more than 1 00 members of AECT participated in crafting this statement of 
definition. The very presence of such a strong statement about philosophy, validated by an esteemed group 
of professionals, underscores the importance of philosophical inquiry in the field. 

Keller (1979) claimed that design needs to address more than practical issues and provide for the 
human spirit. Specifically, Keller referenced Plato’s Republic and described three parts of the soul, which 
includes wisdom or reason (associated with the head), honor or spiritedness (associated with our heart) and 
personal gain (related to appetites). Keller (1979)related behaviorism to controlling individual appetites 
and cognitivism to reasoning abilities, “but with respect to the heart or spirit of the learner, which 
represents individual determination and persi stance, we lack an adequate, systematic approach” (p. 27). A 
few years later, Keller (1 983) reinforced the idea that motivation is the forgotten heart of instructional 
design. 

In 1982, Thomas Luiz (1982) conducted a philosophical investigation of educational and 
instructional practices and techniques entitled “A Comparative Study of Humanism and Pragmatism as 
They Relate to Decision Making in Instructional Development Processes.” Luiz (1982) sought to answer 
the question whether a philosophical inquiry would provide a framework for enabling instructional 
developers to make better and more consistent decisions. Luiz (1982) posited that 

“Instructional developers need to know the implications of their decisions when 
advocating one philosophy, rather than another. It is assumed that their personal 
philosophies, implicit in their actions, act as a screening device through which their 
individual decisions are filtered” (p. 110). 

Luiz (1982) concluded that an instructional developer’s philosophical orientation served as a 
device that filtered instructional development decisions, a perspective that lends support to Ely’s notion of 
philosophy. 

In 1983, Koetting (1983) explored the notion of knowledge in instructional technology and 
developed an epistemological framework for inquiry in our field. Koetting’s paper, “Philosophical 
Foundations of Instructional Technology,” (1983) may be one of the first works to directly relate critical 
theory to the field of instructional technology (Nichols & Allen-Brown, 1 996). Based upon the assumption 
that Instructional Technology is rooted in an empirical view of knowledge, Koetting (1983) discussed the 
implications for future research; and, proposed alternative philosophical and theoretical frameworks for 
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inquiry within the field. Next to Ely’s paper in 1 970, Koetting’s paper was one of the few specific papers 
that connected philosophy directly to the field of instructional technology. 

Despite Luiz’s novel contribution to the Instructional Technology theory base, and Koetting’s 
early exploration of critical theory, philosophical inquiry has since remained relatively dormant in the field. 
By 1991, Koetting and Januszewski (1991) concluded that philosophical debate was relatively absent from 
the instructional technology literature and suggested that . there is not much work that looks at differing 
conceptualizations and frameworks within educational technology” (p. 2). However, the emergence of 
post-modern and constructivist orientations in instructional technology may be breathing new life into 
philosophical inquiry within the field. Denis Hylnka (1992), who has published widely on post- 
modernism, asserts that “any philosophy which can help us to illuminate what we do, how we do it, and 
why we do it, is worth our time and our effort” (p. 4). 

In 1993, Ertmer and Newby presented two opposing positions on the origin of knowledge - 
empiricism and rationalism - illustrating the philosophical origins of behaviorism, cognitivism and 
constructivism. Albeit brief, these authors made a clear connection between current learning theories and 
their historical foundations in philosophy (Ertmer & Newby, 1993). 

By 1 996, The Handbook of Research for Educational Communications and Technology (Jonassen, 
1 996), devoted an entire chapter to “Philosophy, Research, and Education” written by Koetting. Like 
Snelbecker (1974), Koetting (1996) asserted that . .theories are derived from philosophies and ideologies” 
(p. 1 142) and he suggests that one’s theoretical stance affects practice, and vice versa (whether one is aware 
of it or not). Interestingly, Koetting’s (1996) more recent perspectives, as reflected in his chapter, are 
similar to Ely’s early concept of philosophy as a “filter” through which decisions are made. 

More recently, Richey (1998) addressed the relationship between practitioner values and the 
perceived relevance of research claiming that “values influence whether the research commands attention 
and how the research problem is defined” (p. 9). While values may be seen as only one element of 
philosophy, the implications address similar issues: one’s orientation to the world exerts influence over 
one’s theoretical stance. 

Finally, philosophy is commanding greater attention in instructional technology textbooks. For 
example. Smith and Ragan (1999) address three philosophical perspectives of instructional designers. 

These “basic types” of philosophy include 1) rationalism (constructivism) which posits that reality is made 
and not found and that reason is the primary source of knowledge, 2) empiricism (objectivism) which 
postulates that experience is the only source of knowledge and reality is objective, and 3) pragmatism, a 
“middle ground” between rationalism and empiricism because knowledge is constantly being negotiated 
based upon changing experiences (Smith & Ragan, 1999). Smith and Ragan’s section on philosophy also 
supports an apparent trend in the literature that acknowledges the personal nature of philosophy as it relates 
to designer decision-making. 

Potential Contributions of Philosophical Inquiry to Instructional Technology 

Philosophical inquiry - the importance and uses of critical thinking - is ill defined in Instructional 
Technology. Criticism could be considered as a method of critical thinking which “. . . links with all other 
paradigms for inquiry being informed by results from other methods and in turn informing other methods 
with different theoretical perspectives” (Belland, Duncan, & Deckman, 1991, pp. 151-152). At its finest, 
criticism in the instructional technology is like art or literary criticism, sharing observations from unique 
vantage points. At its worst, criticism can be ugly or unkind; a blatant disregard for common courtesies. 

The information that follows is offered as a veritable “starting point” or framework for philosophical 
inquiry in the field. Here, a utilitarian and commonplace usage of philosophy (Ely, 1970) will serve as a 
guiding principle for discussions about philosophical analysis and four dimensions of philosophical inquiry. 

Philosophical Analysis 

For the instructional technologist, analysis is fuel for design activities. Whether analysis is a 
discrete phase, or fully integrated into design, analysis is generally a reductionistic activity, breaking 
information down into its basic elements. For example, Job/Task Analysis may explore a job function in 
terms of general job duties, which breakdown into specific tasks, which breakdown into specific task 
elements. Similarly, philosophical analysis is the process of taking ideas apart and putting them back 
together in order to understand how they function (Morris, 1 999). Morris (1999) notes that “the analysis of 
knowledge as properly justified true belief breaks it down into what are called necessary and sufficient 
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conditions, or, to be more exact, into individually necessary and jointly sufficient conditions” (p. 45). 
Accordingly, in order for knowledge to exist, belief is necessary. Philosophical analysis, therefore, is a 
process for examining beliefs. 

The concept of learning may be used to illustrate the process of philosophical analysis, as it relates 
to instructional technology. To analyze learning philosophically, one must first break it down into 
individually necessary and jointly sufficient conditions. The first step is to begin with a definition of 
learning: “A relatively stable change in knowledge or behavior as a result of experience” (Mayer, 1 982). 
The next step is to identify each component of the definition: knowledge, behavior; experience. According 
to this definition, there appears to be three conditions and each is necessary for learning to occur. You 
can’t have learning unless there’s a stable change in knowledge or behavior and it must result from 
experience. However, these conditions alone aren’t sufficient. For example, someone could be teaching 
manipulative math and the desired outcome would be a demonstration of this capability. If an individual 
can calculate math problems in his head, then we would have no idea if learning occurred. Therefore, 
knowledge or behavior must also be observable, another condition. But, the person we’re observing could 
be a savant or biologically gifted and his prowess with math may not result from experience, so we still 
don’t have sufficiency. Is it possible that without an observer, not only does the behavior not exist, but 
neither does the experience? Perhaps one more condition is necessary. Perhaps we need to explore our 
definitions of knowledge and behavior. The point to be made is that each one of our conditions is 
necessary, and all of them jointly would be sufficient for learning. This process of philosophical analysis 
sheds light upon our understanding of important concepts - and beliefs - that are intimately related to 
instructional technology. Philosophical analysis is one mode of inquiry; yet, the central concern of 
philosophy - how to think critically - could be framed according to four general areas of inquiry known as 
“transcendentals” (Morris, 1 997). 

The Four Transcendentals — A Framework for Philosophical Inquiry 

How frequently do we examine our beliefs? If we’re like most people, instructional technologists 
are too busy living their lives to stop and think about beliefs. But, the impact on instruction and the field 
could be far-reaching. There are four “transcendentals” or timeless values that span across all manner of 
objects (Morris, 1 997) which provide a foundation for philosophical inquiry in the field of instructional 
technology. These dimensions of human experience include: 

1 . The Intellectual Dimension, which aims at Truth 

2. The Aesthetic Dimension, which aims at Beauty 

3. The Moral Dimension, which aims at Goodness 

4. The Spiritual Dimension, which aims at Unity (Morris, 1 997, p.l 9-20). 

The following section will provide a brief overview of each of these values and illustrate their 
potential relevance to the field of instructional technology. 

L The intellectual dimension, which aims at truth. If we accept the notion that human beings 
think and that all people have a mind, then we must also recognize that all people have an intellectual 
capacity. The intellectual dimension reflects the pursuit of knowledge. As noted earlier, knowledge can’t 
exist without belief because a person can’t know something without first believing in it. Since false beliefs 
also exist in our world (c.g., I can believe that the sky is falling), the intellectual dimension, which aims at 
truth, serves as a guide to reality by clarifying the way things are. The concept of truth, as it relates to 
instructional technology, is a contemporary issue in the field because it has become a point of departure 
among the more recent theories of learning. Instruction that is based upon behavioral or cognitive theories 
of learning assumes that truth is objective. In other words, the purpose of instruction is to achieve defined 
learning objectives or to acquire defined areas of content. Constructivism, which may be defined as either 
a general theory of cognition (Wilson, 1997) or a philosophical orientation (Lebow, 1995) is centered on 
the assumption that truth and reality are subjective. Constructivists define learning as an active process of 
constructing knowledge rather than acquiring it (Duffy & Cunningham, 1996). Understanding one’s own 
beliefs about truth can help the instructional technologist manage decisions during the instructional design 
process, as well as prevent or resolve problems along the way. In this way, philosophical inquiry could 
guide the instructional technologist toward action that is aligned with her worldview or call attention to any 
inconsistencies. 
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2. The aesthetic dimension, which aims at beauty. The aesthetic dimension of human experience, 
which seeks beauty, encompasses all forms of delight (Morris, 1999). Before discussing the aesthetic 
dimension in relation to instructional technology, it may be helpful to pause for a moment think about how 
you feel when you encounter something beautiful. Whether it’s a work of art, a mountain top or a child’s 
face, beauty liberates, refreshes, restores and inspires (Morris, 1997). In this regard, beauty can be 
something experienced or felt by the learner; or it may simply be finding an elegant solution to an 
instructional problem. If . . art, craft, and science all have a role to play in the strategies and tactics of 
instructional development” (Davies, 1981/1991, p. 96), then 

the truly creative scientist needs something of the artist’s divergent thought to see new 
possibilities while for his part the artist needs to be able to apply the single-minded 
perseverance of the scientist to develop his ideas. What makes design such a challenging 
task psychologically is the very even balance of these two sets of mental skills that are 
needed to produce creative work (Lawson as cited in Rowland, 1993, p. 86). 

Philosophical inquiry in the aesthetic dimension would most likely explore the nature of design. 
Like the architect, the instructional designer can build instruction that serves a purpose while the true artist 
will find ways to brighten the human experience. 

3. The moral dimension, which aims at goodness. Just as truth and beauty enhance life, goodness 
enriches human experience. Overstreet (as cited in Morris, 1 997) provides insight into the way in which 
these three transcendentals seem to be connected: “Goodness is a special kind of truth and beauty. It is 
truth and beauty in human behavior” (p. 1 16). When people “do the right thing,” inner strength and 
interpersonal strength is nurtured and developed. For the instructional technologist, the question is what is 
the right thing to do? 

Although some people believe that morality is concerned with private values and ethics deals vrith 
public conduct, the terms “ethics” and “morality” can be used synonymously because they both reflect the 
same fundamental values (Morris, 1 997). Ethical conduct is a contemporary issue in the field that has been 
addressed by the International Board of Standards for Training, Performance and Instruction (ibstpi, 1999). 
The IBSTPI Credo for Performance Technologists and Instructional Designers addresses the notion of both 
personal/professional development and social responsibility. The statements that comprise this credo 
reflect a consensus from Delphi Respondents about “doing the right thing” in our field. 

Philosophical inquiry about designer decision making offers another way to explore the moral 
dimension. In a milieu where instructional technologists are continually challenged to be better, faster and 
cheaper (Tessmer & Wedman, 1990), questions about the nature of ethical decision making become 
paramount. At the most basic level, philosophical inquiry would explore whether or not our decisions are 
compatible with our belief systems. This approach would require awareness of one’s ovm beliefs, as well 
as an understanding of what the client or sponsor believes and values. One clear implication for the 
instructional technologist is that it may be necessary to work with clients or sponsors to identify or discover 
their beliefs. 

As Luiz (1982) noted, designers interpret instructional problems through the beliefs and values 
that comprise their personal philosophy; yet, in most situations, a range of choices often exists. Therefore, 
the first decision one makes would be a judgement about which feasible options to select, followed by a 
decision to act on one of the choices (Morris, 1997). The path to ethical decision making — in general — is 
often defined by “rules for living” which, for the instructional technologist, could be found in a 
professional credo, a procedural model, personal philosophy or any combination thereof Along the way to 
ethical decision making, instructional technologists can stop and ask oneself if his/her decisions are aligned 
with one’s beliefs. 

Another mode of philosophical inquiry in the moral dimension, could be a “moral audit” of 
instructional decisions that have been made that were later regretted. Here, the instructional technologist 
could explore any consistent themes and/or consequences that resulted from decisions that were morally or 
ethically wrong (decisions that were incongruous with personal beliefs). The ultimate benefits of this 
activity would be wisdom and virtue, a perspective on building inner and interpersonal strength, and most 
importantly, insight into the impact of our actions on the learner. 
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4. The spiritual dimension, which aims at unity. One facet of the spiritual dimension of human 
experience is concerned with going beyond the surface and exploring a deeper view of the world. The 
notion of depth, as it relates to the spiritual dimension, is really about transcending surface realities and 
looking beyond what meets the eye. For the instructional technologist, the spiritual dimension involves 
looking for hidden details and revealing layers of meaning; looking beyond the observable. The challenge 
is to view instruction like one would experience sculpture or art, seeing new details from every new 
vantage point. 

Spirituality in our field would be concerned with more than just solving instructional problems, or 
conducting valid and reliable research, which are at the surface; for the instructional technologist, 
spirituality is all about learning and reflection. It’s about deep learning that results in irreversible change 
(Senge, Kleiner, Roberts, Ross, & Smith, 1994), reflecting on the experience, as well as the capacity to 
share the depth of one’s perspective with others. A good illustration of this concept is self- reflexive 
research which “... involves professional self-critique, in which the researchers own up to their values and 
how they are present in their work as interested people” (Anderson & Damarin, 1996, p. 273). 

Another facet of the spiritual dimension deals with connectedness, which seeks unity: 

. . . connectedness, or intimate integration, between our thoughts and our actions, between 
our beliefs and emotions, between ourselves and others, between human beings and the 
rest of nature, between all of nature and nature’s source. Unlimited connectedness. 
Ultimate unity. (Morris, 1997, p. 179) 

Issues of integration in the field have been addressed by Seels (1995) who expressed “. . . a need to 
expand ID [instructional design] fundamentals and to resolve many issues by integrating theories and 
integrating theory and practice” (p. 247). Interestingly, the importance of connecting and integrating the 
arts and sciences appears to be a high priority (Seels, 1995); and, the very stuff of which philosophical 
inquiry is made. Finn (1953/1996) once remarked that clarification was one of the jobs of the philosopher. 
The issues of integration in the field could be among the greatest challenges for contemporary thinkers in 
instructional technology today. 

Another way to understand connectedness might be to explore relationships among designer, 
instructor, delivery system, learner and the environment. Here, the role of context in learning and 
instructional design (Tessmer & Richey, 1997) provides a newer path of inquiry into the spiritual 
dimension. Context is all about depth and connection. While this discussion about the spiritual dimension 
may appear to be more academic than practical, Tessmer and Richey (1997) offer the instructional 
technologist a variety of tools and techniques for dealing with issues of depth and connectedness in 
instructional design. For the philosopher, it is also important to ask questions about why the tools and 
techniques of contextual analysis are important. While the familiar portal of systems theory helps explain 
complexity and the overall importance of connectedness in instruction (and our world), it often falls short 
of explaining matters of the spirit. Here, philosophical inquiry can uncover deeper layers of meaning. 

While there doesn’t appear to be much discussion about “freeing the spirit” in instructional 
technology, Keller’s (1983) claim that motivation is the forgotten heart of instructional design suggests that 
design needs to address more than practical issues and provide for the human spirit (Keller, 1979). This 
section on the spiritual dimension will conclude by posing questions about four spiritual needs (Morris, 
1999) that have implications for instructional technologists: (1) How can we help the learner feel special, 
distinctive and unique? If we all have a need to feel important, how can instruction facilitate this basic 
spiritual need? (2) How can we help the learner feel connected to something important? If people yearn for 
a feeling of belonging, then how can we build a sense of spirit into instruction? (3) How can we help the 
learner feel like he/she is making a difference? If people have spiritual needs about feeling useful, then 
how can we facilitate experiences that are fulfilling and meaningful? (4) How can we help the learner 
develop a deep understanding and passion for what’s important? If people have a deep spiritual need to 
understand how their efforts fit into the “big picture,” then how can we help people see that instruction is 
surrounded by a multiple of factors? 

These questions address four spiritual needs that could potentially be nurtured by the instructional 
technologist: Uniqueness, Union, Usefulness and Understanding (Morris, 1999). At the very least, they 
provide more fodder for philosophical inquiry in the spiritual dimension. 
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Conclusion 



Philosophy, as method of inquiry, appears to be making a comeback (Anderson, 1995). The 
philosophical counseling movement, which combines the wisdom of the world’s greatest thinkers with 
practice is growing (Marinoff, 1 999), the search for spirit in the workplace (Lee & Zemke, 1 993) and the 
notion of “corporate soul” (Morris, 1997) illustrate a new era in which we find ourselves living, . a time 
of rethinking and rebuilding in which beliefs about belief are shaken as never before, a time in which issues 
once left to the philosophers - such as the nature of truth - become matters of vital everyday importance to 
ordinary people” (Anderson, 1 995, p. 3). And, in our field, . . which draws knowledge and practice from 
a wide range of arts and sciences, educational technology should be able to use a variety of ways of 
investigating and knowing in order to guide inquiry and practice” (Belland et al,, 1991, p, 151.). Thus, 
philosophy is a valid method of inquiry in instructional technology (Koetting, 1996). 

Although often mistaken as an “ivory tower” academic discipline, philosophy can provide 
guidance for the instructional designer and potentially improve decision quality and speed. People have 
beliefs; values, which are beliefs of relative importance; and, philosophies that are composite statements, 
which guide and direct their behavior. For this reason alone, philosophy is a valid mode of inquiry in our 
field. However, theory is derived from philosophy (Koetting, 1996; Smith & Ragan, 1999; Snelbecker, 
1974); therefore, philosophical inquiry offers deeper insight into the theoretical foundations of our field. 
And, our field advances through interactions between theory and practice, and philosophical inquiry builds 
fundamental skills that are valued in our field. 

At a deeper level, the voices of our founders can help us develop an appreciation for philosophy in 
instructional technology because it is their wisdom that set the course for the field as we know it today. 
Their vision appears timeless. It only makes sense that “.. .the philosophical lineage of Charters-Dale- 
Hoban-Finn ... yielded the most productive thinking about the field ...” (Ely, 1970, p.85). One could 
simply look at the philosophical lineage of Socrates - Plato - Aristotle to recognize the tremendous impact 
of teacher-student relationships. Ely (personal communication), who has served as a veritable conscience 
for the field, was greatly influenced by Dale and Finn; and. Seels, whose concerns about the field include 
definition (Seels & Richey, 1994), theory development (Seels, 1997) and issues of integration (Seels, 

1995) , was Edgar Dale’s last doctoral student. Wisdom, practical insight for living, is timeless. 

Philosophy is linked to all modes of inquiry by virtue of its fundamental and timeless question, 
“why?” Through progressive layers of questions and thorough examination, philosophical inquiry 
produces wisdom. Wisdom is worthy of pursuit, not only because it helps us live good lives, (Morris, 
1999), but it also helps us become better researchers and designers. Posing questions is the major task of 
philosophy, which is the basis for research and the foundation upon which our field is built (Koetting, 

1996) . Simply put, “without good questions, there is no inquiry” (Koetting, 1996, p. 1 144). 

The role of philosophy in the field of Instructional Technology has not been explicated. 
Exploratory studies that investigate philosophical assumptions about the theory and practice of 
Instructional Technology are rare. Given the variety of philosophical orientations that currently exist in the 
field today, this paper may provide insight into understanding that various ways in which philosophy 
shapes instructional design practice. 
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Research Issues of Motivation in WBI 
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Abstract 

The importance of learner motivation for successful WBI has been recognized in the field of 
educational technology. However, we have not accumulated a substantial knowledge about 
motivational issues in WBI. This paper proposes a definition of motivation to learn in WBI and 
systematically describes three types of motivation. Then, major motivational problems in WBI are 
explained. Finally, research issues of motivation in WBI are suggested. 



Introduction 

World Wide Web (WWW) is becoming more popular as a platform of instruction. However, we 
have not accumulated substantial theories or principles to use in the design and development of effective, 
efficient, and appealing web-based instruction (WBI). Especially, few systematic discussions of learner 
motivation associated with web-based instruction are found in literature. Therefore, there is a need to 
explore motivational issues and suggest critical areas of research on motivation in WBI. 

Motivational Features of WBI 

WBI may be defined as a type of distance instruction that utilizes the diverse resources of the 
internet based on computers. This implies that WBI is a derivative of both traditional distance education 
and stand-alone computer-assisted instruction. Therefore, it may have the same motivational features that 
have been encountered by distance educators and computer-assisted multimedia instruction designers. For 
example, teachers and learners are separated in time and space. This means that learners may feel isolated, 
resulting in high dropout rates. The use of fancy multimedia effect will disappear as learners become more 
familiar with computers and the internet. 

However, WBI has one unique feature that both distance education and stand-alone CAI do not 
have. It is the fact that WBI allows instructors and students to interact synchronously or asynchronously 
through the computerized networks. This implies that WBI may not have the same problems that faced 
distance educators or stand-alone CAI designers. We need to study how to utilize this feature to design 
more motivating WBI. 



DeHnition of Motivation on WBI 

Researchers have emphasized the importance of learners’ motivation for successful WBI 
(Duchastel, 1997). However, just emphasizing the importance does not ensure motivating WBI. What 
instructional designers need is a practical definition that will guide them in designing motivating WBI. In 
the literature of educational psychology, there have been many definitions suggested by different 
psychological perspectives (Stipek, 1 993). But psychologists do not provide a practical definition to be 
used by instructional designers. They tend to contribute to the understanding of the nature of motivation, 
not to provide a prescriptive definition that will guide instructional design of motivating WBI. 

In this sense, Keller’s approach is unique and valuable in that his ARCS model is practical enough 
to be used by instructional designers (Keller, 1 983). The ARCS model is a comprehensive model that helps 
us understand motivation more concretely. In the ARCS model, Keller defines motivation as ‘the direction 
and intensity of behavior’ which is influenced by four categories - attention, relevance, confidence, and 
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satisfaction. With this conceptualization, instructional designers do not need to worry about the complexity 
of the construct of motivation because the four categories give a neat understanding of motivation to 
designers. Consequently, the designers may use that understanding when they design WBI. In fact, in the 
ARCS model, the definition of motivation is practically used in two ways: first, it guides the systematic 
process of motivational design, and second, it suggests motivational strategies for each of the four 
components > attention, relevance, confidence, and satisfaction. 

Now, if we use the ARCS model for the designing of motivating WBI, we may define motivation 
to learn as ‘direction and magnitude of effort to learn through WBI, which can be explained in terms of 
attention, relevance, confidence, and satisfaction.” 

Types and Problems of Motivation in WBI 

Identifying research issues of motivation in WBI requires us to systematically analyze the process 
learners go through WBI. Imagine learners who enter WBI. They will first need to decide whether they 
will participate WBI or not. The motivation to influence this decision may be called as ‘motivation to 
initiate’. Once they have started WBI, they need to decide continually on whether they will persist in the 
instruction or not. The motivation to influence this decision may be called as ‘motivation to persist’. Now, 
when they finish WBI, they will need to decide whether to continue studying the same or or similar subject 
through WBI. The motivation behind this decision would be called as ‘motivation to continue’. (Refer to 
Figure 1) 

Figure 1. Types and problems of motivation on WBI 



External Factors 




No participation Dropouts No continuation 

Non-starters Procrastination Non-referral 



Now, based on the ARCS model, these three types of motivation can be defined as the following, 

1 . Motivation to initiate; direction and magnitude of effort to learn by participating in WBI, 
which can be explained in terms of attention, relevance, confidence, and satisfaction, 

2. Motivation to persist: direction and magnitude of effort to learn by persisting in WBI, 
which can be explained in terms of attention, relevance, confidence, and satisfaction, 

3. Motivation to continue: direction and magnitude of effort to continue learning the same 
or similar subject through WBI, which can be explained in terms of attention, relevance, 
confidence, and satisfaction. 

There are two reasons to conceptualize these three types of motivation. One is that motivation 
tends to change through instruction (Coldeway, 1991 ; Song & Keller, 1999). The other is that reasons for 
participation may be different from those for dropping out and continuation (Wilkes & Burnham, 1991). 
That is, factors (personal factors, internal WBI factors, or external WBI factors) influencing learners to 



330 



329 



participate, persist, and continue would be different. Therefore, with this conceptualization, instructional 
designers can be better prepared to address motivational problems at each stage of learning process. 
Belawati (1998) suggested two goal commitments of learners in distance education: one is initial goal 
commitment, the other is modified goal commitment. Motivation to participate is related to initial goal 
commitment, while the other two types of motivation are more related to modified goal commitment. 

Major motivational problems of WBI can be explained in relation with these three types of 
motivation. When learners do not have proper motivation to participate, they will not participate or 
become non-starters (Coldeway, 1991). In distance education, non-starters are the ones who register for the 
course but never start their learning. It is reported that the main reason many non-starters drop the course is 
not because of the quality of the course. 

When learners do not have proper motivation to persist, they will drop the course or they will 
procrastinate. Dropping out from lack of motivation to persist is different from that of non-starters. If 
learners who are not non-starters drop out, the quality of WBI should be questioned. That is, WBI is not 
motivating. Maybe, interactions (instructors-leamers, content-learners, learners-1 earners interaction) 
provided to learners are not motivating. Procrastination is another problem. WBI instructors need to be 
responsive to any of the indicators of procrastination. However, we have not accumulated any knowledge 
about it. 

When learners do not have proper motivation to continue, they will go away or they will not 
recommend WBI to others. Continuing motivation (Maehr, 1976) is basically intrinsic motivation. 

Learners having intrinsic motivation study hard because they like the learning itself and they tend to study 
more about the same or similar topics. Instructional designers should consider how the WBI they design 
can increase learners’ continuing motivation. Otherwise, they will encounter the problem of non-referral 
by learners. 

Conclusion: Research Issues of Motivation on WBI 

Then, what are the research issues of motivation in WBI? There are 7 categories of motivational 
issues that need to be further studied. The listings are not intended to be exclusive. However, research on 
these issues will provide building blocks for designing a motivating WBI. 

Category 1 : What influences motivation to initiate? 

Who prefers WBI? 

What types of WBI are preferred? 

What are the barriers to registration (participation)? 

How to measure motivation to initiate? 

Category 2: Non-starters? 

What percentage of enrolled students become non-starters? 

What are indicators of non-starters? 

How to deal with non-starters? 

Category 3: Dropout rates, procrastination? 

What is the model to explain dropouts on WBI? 

What are indicators of procrastination? 

How to measure motivation to persist? 

How to increase motivation to persist? 
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Category 4: Interaction? 

What motivational strategies should be provided for each type of interaction (instructors-learners, content- 
learners, leamers-leamers interaction)? 

What combination of interaction types should be provided? 

Category 5: Web Features and motivation? 

Which individual differences (learning style etc.) are more appropriate to WBI digital environment? 
Should we provide more multimedia or less? 

Category 6: Motivation to continue? 

How to measure motivation to continue? 

How to increase motivation to continue? 

What is the relationship between motivation to continue and the other two types of motivation? 

Category 7: Need for comprehensive model? 

What is the model to describe motivation in WBI? 

What is the model to prescribe motivation in WBI? 

How to integrate ISD model and Motivational Design Model (such as ARCS model) for designing WBI? 
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Abstract 



Thirty-one undergraduates read a 204-word passage describing social groupings of fish and their 
color, depth, size and diet. Subjects concurrently viewed a graphic organizer (GO) that presented 
this conceptual information in a matrix-like format. Half the GOs were semantically congruent 
with the text and organized according to social grouping while the remaining ones were spatially 
organized by fish depth. Subjects answered comparison and inference-type questions immediately 
after the study period and again two days later. Analysis of variance showed semantic congruity 
influenced GO efficacy but only for delayed recall of material related to the main topic of the text. 

More than ever before, people need assistance in absorbing and making sense of the enormous 
flow of data generated by the Information Age. Consequently, there has been increased interest in the 
potential of graphics for assimilating and mentally processing knowledge. In some cases, graphics 
facilitate acquisition of new knowledge in a supplemental way by embellishing or illustrating an 
accompanying text passage (Levie & Lentz, 1982; Levin & Lesgold, 1978). Alternatively, graphics may 
offer an meaningful context that is critical to how well an accompanying text is interpreted and 
remembered (Bransford & Johnson, 1972; Schallert, 1980). 

Sometimes a graphic display may be preferred over text for delivering information simply because 
the former possesses a cognitive capacity (Kozma, 1991) lacking in the latter. For example, Larkin and 
Simon (1987) persuasively demonstrated that a simple diagram of a pulley was vastly more effective than 
its sentential counterpart in helping one to understand the physics principles being presented. Because, 
unlike text, the processing of diagram-based information is not serially constrained (Bauer & Johnson- 
Laird, 1993), it offers a “computational efficiency” that assists in determining the relationships of 
components depicted. 

Another type of display that may be an efficient alternative to text is the graphic organizer or GO 
(pronounced “Gee-Oh”). This is a tabular array of categorical information that is indexed over one or more 
variables. Consider, for example, a text passage about types of fish and the characteristics of each such as 
size, color, social grouping (i.e., solitary, small, or school), swimming depth, and diet. In this case, a 
corresponding GO might arrange fish names in a row according to swimming depth while listing, as a 
column below every name, the social grouping, color, size, and diet of each fish. Such an arrangement aids 
visual reasoning (Gattis & Holyoak, 1996) by using space to organize factual data in a way that facilitates 
quick comparisons between types of fish across several different variables. Because a GO reveals how two 
or more characteristics (e.g., size and color) vary with respect to one another, learners can more readily 
draw inferences based on conceptual interrelationship (e.g., larger fish tend to be lighter in color). 

Robinson and Schraw (1994) studied the aforementioned GO and the 204-word text from which it 
was derived. Subjects read the text and then studied either the GO, an outline of the text, or completed a 
second reading of the text. They then judged the accuracy of 68 separate statements related to the text, 
either comparing two types of fish on a particular characteristic or inferring the relationship between two 
characteristics. Significantly more correct responses were made by subjects studying a graphic organizer 
compared to those rereading the text a second time. 

While GOs have been shown to be effective adjuncts to text, like other types of displays, their 
value can probably be flirther improved through improvements in design. Merely rearranging the verbal 
elements of a display can have a profound influence on how it is used to analyze the information presented. 
Wainer (1992), for instance, demonstrated that use of proximity to spatially cluster the labels of a table can 



335 




evoke deeper processing of the display’s data. Similarly, Schuch, Tada, von Eberstein, and Kealy (1999) 
found that spatially clustering the elements of a GO according to categorical membership facilitated 
significantly better comparative judgments about the material studied. 

Another possible influence on GO efficacy, the focus of the current study, is the “semantic 
congruency” between a text and its corresponding GO when both are studied concurrently. Just as prose 
has a semantic structure-paragraphs begin with a main or topic sentence followed by ones subordinate in 
meaning-so too does a graphic display. How one “reads” both tables and diagrams is determined by the 
visually prominent display of information that cues viewers on how the display is conceptually organized. 

In both the Robinson and Schraw (1 994) and Schuch, Tada, von Eberstein, and Kealy (1999) studies, the 
topic of the experimental text was the social groupings of fish. By contrast, the corresponding GO was 
primarily organized according to the depth at which the six types of fish lived: 200, 400 or 600 feet. This 
contradiction raises the question of whether such a semantic incongruity between text and GO causes 
interference during mental processing along with a resulting decrement in learning. Since the linguistic 
structure of text has a powerful effect on what readers recall (Johnson, 1970), it was reasoned that 
contradictions between the apparent semantic structures of the text and its accompanying GO may impede 
learning performance. By contrast, when the superordinate conceptual organization of the GO, reflected by 
the categories used for the top row of the display, were consistent with the structure of the corresponding 
text, we speculated that the GO would yield superior learning. 

Method 

Design and subjects 

The study consisted of a 2 Semantic Organization (Congruent vs. Incongruent) x 2 Question Type 
(Comparison vs. Inference) x 2 Recall (Immediate vs. Delayed) x 5 Topic (Color vs. Depth vs. Feeding vs. 
Grouping vs. Size) factorial design. Semantic Organization was varied between-subjects while the 
remaining variables served as repeated measures. Thirty-one undergraduates participated in the study, 
receiving course credit for doing so. 

Materials 

Using the same text as that used in the Robinson and Schraw (1994) study, two versions of a GO 
were created: one organized by fish depth (200, 400, and 600 feet) and the other by social grouping 
(solitary, small, and school). The resulting GOs, shown at the top and bottom of Figure 1, respectively, 
were incorporated into two separate computer programs developed using Authorware. When run, the 
programs presented subjects with instruction on graphic organizers, showed samples of the comparison and 
inference questions they would later complete, and provided practice on the experimental task. The 
programs also collected and stored the responses by subjects to test items as well as the latencies of these 
responses. 
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Fi£ure 1 . Semantically Congruent (Top) and lncon2nient (Bottom) Graphic Orsanizers Used in the Study 
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Computer displays were presented on 15-inch monitors at SVGA resolution (600 x 800 pixels). 
All computer screens were designed with a light green background and dark blue lowercase 14-point Ariel 
typeface with the exception of instructional examples and GO labels that used black 10-point lettering. 

Since some subjects would likely be unfamiliar with graphic organizers, the experimental 
programs contained six screens that explained the nature and purpose of a GO and showed an example 
using facts about different buffalo (see Figure 2). By directing the viewer to specific points in the GO and 
text, the instruction illustrated how the GO could be especially useftil for making comparisons and 
inferences from the information. During the instructional phase, subjects could progress or review the 
information by using the mouse to click the forward and back buttons, respectively, at the bottom of the 
screen. 

Figure 2. Example of Instruction on the Use of GOs Contained in the Computer-based Experimental 
Treatments 



For example, notice the excerpt of a story below about 
various types of buffalo. To the right is an accompanying 
G-0 that spatially organizes facts from the text in a way that 
makes the categorical relationships in the story evident. 

As you can see in this example, a G-0 is like a matrix or 
table that arranges categorical inforrhation, like "weight" or 
"lifespan" by rows and columns. 




LIFESPAN OF BUFFALO 



\ African Asiatic Water 
Buffalo . Buffalo 



25 years 30 years 



2000 lbs. 



5 ft. 



2500 lbs. 



4 ft. 
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The instruction explained that, while reading the text about fish types that would follow, subjects 
could view the accompanying GO at any time by holding down the mouse button. This allowed subject to 
read the text while periodically consulting the GO whenever they wished to do so. In the Robinson and 
Schraw (1994) study subjects first read the text passage, then studied the GO only after the text had been 
removed. By contrast, we chose to give subjects the means to study the text and GO concurrently, 
reasoning that this approach created a learning environment that more closely approximated the way people 
actually use a graphic organizer. 

Another feature of the current study that was a departure from the Robinson and Schraw (1994) 
experiment was the nature of the criterion measure used. In the Robinson and Schraw study, subjects 
responded to 30 comparison and 30 inference statements, half within each category true and the rest false, 
to which they responded by pressing either of two keypads. However, a replication of this methodology by 
Schuch, Tada, von Eberstein, and Kealy (1999) showed only chance differences between subject responses 
to false comparison and inference statements. 

The current study used a different strategy whereby subjects saw comparison and inference items 
in a two-item multiple-choice question format. An example of a comparison question used was, “Which is 
darker in color?” (Tin or Arch) while a sample inference-type question was, “Smaller groupings of fish 

tend to be in size.” (bigger or smaller). Although this approach also involved a 50 percent chance 

or making a correct response, it cut the total number of question in half thereby reducing the possibility of 
subject fatigue while potentially evoking greater semantic processing through a use of a more authentic 
recall task. 

Both the pool of 1 5 comparison items and the group of 1 5 inference items consisted of three 
questions on each of five fish characteristics depicted in the text and GO: color, depth, social grouping, 
diet, and size. For a given characteristic such as color (e.g., black, brown, blue, orange, yellow, white), an 
attempt was made to select attributes that were not mentioned in any of the remaining questions. While not 
exhausting all the possible questions that could have been formed, this procedure nevertheless provided a 
broad and balanced coverage of the content studied by subjects. The entire pool of 30 comparison and 
inference questions thus constructed were built into the computer-based treatments so that they were 
presented in a separate random order to each subject during the experiment. 

Procedures 

Subjects participated in an experimental session in groups of about ten. Upon arrival for the 
session, each subject was randomly assigned to a desktop computer that contained one of the two 
experimental conditions. This resulted in a total of 1 8 subjects assigned to the Congruent treatment and 1 3 
assigned to the Incongruent treatment. 

Once all subjects were seated in front of their blank computer screens, the experimenter explained 
that they would be examining a short text passage and related display and then asked questions about what 
they studied. After all procedural questions were answered, subjects pressed the TAB key to start the 
program which prompted them to type their name and press the ENTER key. 

Subjects then viewed the first of the six instructional screens which repeated that they would have 
five minutes to study a 200-word text about different types offish. To assist them in monitoring the 
passing time, the instruction stated, a small clock-like icon similar to the one currently on their screen 
would appear in the upper-left comer of the text. Proceeding at their own pace, subjects read through the 
remaining five instmctional screens. The last screen again informed them that after studying the text and 
GO about fish they would answer comparison and inference types of questions about what they read. This 
time, however, they were shown examples of a hypothetical comparison and inference question based on 
the text about buffalo. Finally, subjects were instmcted to raise their hand if a question remained about the 
experimental task or, if not, to wait for further instmctions. When all questions had been answered, they 
were told to click on the word “please” in the text to progress to the screen containing the experimental text 
and graphic organizer. 

Immediately after the five-minute study period, three two-column simple addition problems 
appeared on the screen. The computer asked subjects to confirm the accuracy of each sum by typing a “Y” 
if correct or a “N” if incorrect. This brief interpolated task was designed to clear the working memory of 
subjects before administering the criterion measure. Upon completion of the three problems, the computer 
prompted them to look up from their screen to indicate they were done. When all subjects indicated 
completion of their arithmetic, the experimenter told them to click on the word “look” to proceed to the 
next task. 
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The screen then showed subjects an example of a hypothetical comparison-type question dealing 
with a fish characteristic not mentioned in the text they studied earlier. At the lower half of the screen 
appeared two shaded rectangles, side-by-side and of equal size, with a one-word answer printed in the 
center of each. Subjects were told to respond to the question shown (“Which fish typically weighs more?”) 
by clicking on the box with the correct answer (either “Cod” or “Dolphin”). Once clicked, both shaded box 
and enclosed word flashed in an inverse tone to signal that that choice had been made. Similarly, subjects 

were next shown an inference-type cloze question (“Fish that weigh less tend to have a lifespan.”) 

and two possible answers each printed in a separate box (“longer” and “shorter”). 

The program informed subjects that an asterisk would appear in the center of the screen for two 
seconds just before each comparison or inference question appeared. They were encouraged to make their 
responses as soon as possible and then told to press the TAB key to practice sample comparison and 
inference questions. After finishing the practice items, subjects were informed that these questions were 
similar to the ones they were about to complete. Subjects were encouraged to pay attention, work quickly, 
and do their best while performing the task. When it was clear that they fully understood what they were 
being asked to do, the experimenter told them to click on the word “best” and to begin the test phase of the 
study. Within both groups of 1 5 questions, three questions dealt with one of the five categories of fish 
characteristics (e.g., depth, grouping, diet, color, and size) reported in the text and GO. The 30 questions 
thus formed were presented during the testing phase one at a time and in random order. Once the last 
person in the room completed answering the questions, all the subjects were dismissed from the study as a 
single group. 

Two days later, subjects reported for a different experimental session (i.e., one they believed 
unrelated to GOs) and assigned to one of the desktop computers. When ready, they pressed the TAB key, 
starting the computer program. Subjects then entered their name and viewed instructions informing them 
that they would be answering the same questions completed two day earlier. Subjects pressed the TAB key 
once again, completed the 30 inference and comparison questions and were dismissed. 

Results and Discussion 

During the testing phase of the study, the experimental program scored a one if a subject’s 
response to an item was correct and a zero if the response was wrong. To facilitate data entry and analysis, 
the computer-based treatments calculated a mean score for each cluster of three test items involving the 
same fish characteristic for both comparison and inference questions. This produced ten mean proportional 
scores, five for comparisons and five for inferences, representing all the five categories of fish 
characteristics mentioned by the text and GO. Hence, a value of zero, .33, .67, or 1 .0 was possible for each 
mean proportional score thus calculated. Following the recommendations of Winer (1 971), an arcsine 
transformation was performed on each of the proportional scores prior to statistical analysis. An alpha 
level of .05 was used for all statistical tests. 

Accuracy of Recall Performance 

Data on the accuracy of subjects’ responses to comparison and inference questions were entered in 
a 2 Semantic Organization x 2 Mental Operation x 2 Recall x 5 Topic repeated measures ANOVA. The 
analysis revealed a significant main effects for the Mental Operation, F(1 , 29) = 4.89, p = .035, and Topic, 
F(4, 1 16) = 3. 12, p = .01 8, variables and a significant Mental Operation x Topic interaction, F(4, 116) = 
4.81, p< .001, that accounted for most of the variability (r|^=.14) in the dependant measure. Figure 3 
depicts the mean proportional scores involved in the Mental Operation x Topic interaction. This illustrates 
the superior performance by subjects on comparison items compared to inference questions and replicates 
the findings of previous research by Robinson and Schraw (1994). Both the role of Topic on performance 
and its interaction with Mental Operation is less clear. Evidently subjects performed poorer on questions 
related to fish depth but only when the questions dealt with comparisons. 
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Mean Proportional Correct 



Figure 3. Influence of Text Topic on Performance of Subjects in Making Comparative and Inferential 
Judgments 
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It was puzzling that subjects' performance showed no significant decrement between the first 
experimental session and the second trial two days later. To examine this further and improve 
interpretability of the data, a separate ANOVA was performed for each of the sessions. This revealed that 
differences between performance on comparison and inference items were significant for immediate, F(l, 

29) = 4.71 , 2 = .038), but not delayed, recall. The Topic variable, on the other hand, only had a significant 
influence on performance occurring two days after study of the GO. The analysis of delayed recall showed 
an interaction between Semantic Organization (i.e., the type of GO studied) and Topic that was not 
significant, F(4, 116) = 2.09, p = .063), by conventional standards, but one that was nevertheless 
interesting. As Figure 4 illustrates, the mean proportional score on comparisons and inference questions 
was notably lower for items dealing with fish grouping when the GO used was spatially organized by 
depth. 

Figure 4. Relationship Between Text Topic and the Semantic Consruity of a Graphic Organizer on Delayed 
Recall 

Effect of a Semantically Incongruous Graphic Organizer (GO) 
on Delayed Performance for Making Comparisons and 
Inferences on Information in an Accompanying Text. 




Text-Display Semantic Congruity 
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While this was the expected outcome, it raises the question of why such a decrement in 
performance by subjects studying a semantically incongruous GO only occurred two days after the display 
was studied? One conceivable explanation is that, due to the dominance of subjects* verbal processing 
abilities, the deleterious effect of an incongruous graphic display had little impact on immediate recall. 
However, two days later when memory of the information was diminished, subjects began to more actively 
use the spatially-encoded GO as a secondary retrieval cue for related information in verbal storage (see 
Kulhavy, Stock, & Kealy, 1993 for a discussion on the value of maps as secondary retrieval cues). 
Hypothetically, if the GO brought into working memory was semantically incongruous with the text, it 
proactively interfered (Osgood, 1 949) with verbal recall — especially information contained in the 
macrostructure of the text (i.e., facts about depth). 

Latency of Recall Performance 

Latencies of subjects* responses to questions were entered into a 2 Semantic Organization x 2 
Mental Operation x 2 Recall x 5 Topic repeated measures ANOVA. This revealed a significant main effect 
for Mental Operation, F(1 , 29) = 24.61 , £ < .000, which accounted for nearly half of subjects* variability 
(r|^= 46) in scores. ANOVA also identified significant main effects for Topic, F(4, 116) = 4.29, £ = .003, 
and, unlike the analysis of test performance. Recall F( 1 , 29) = 1 3.67, £<.001. 

Additionally, ANOVA indicated significant two-way interactions for Semantic Congruity x 
Mental Operation, F(U 29) = 4.93, £= .034, Semantic Congruity x Topic, F^^, 116) = 2.67, £= .035, and 
Mental Operation x Recall, F(1 , 29) = 8. 14, £ = .008. Figure 5 depicts the relative effect of Semantic 
Congruity on latency of performance for comparison and inference questions. As expected, subjects took 
much less time in responding to the comparison questions compared to the relatively more difficult 
inference items. On the other hand, the dramatic drop in latency of response among subjects who studied 
the semantically incongruous GO, may be an indication of mental processing that was simply more 
superficial rather than more efficient. 
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Figure 5. Effect ofText-to-disvlav Semantic Consruencv on Response Latencies forAnsweriri2 Comparison 
and Inference Questions 
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This hypothesis is supported, in part, by Figure 6 which shows the differential effects of GO type 
on speed for processing questions according to the Topic variable. The slower processing speed evident for 
questions related to grouping, the main topic of the text, points to the dominant status of verbal processing 
among adult learners. The semantic structure of text provides a strong interpretive mechanism which, in 
the present case, signaled that the main topic of the passage was the socialization of fish. Given the 
absence of explicitly stated educational objectives, learners are likely to use the semantic structure of a 
studied text to help determine the instructional task they will be asked to perform. The relationship 
between instructional objectives and task expectations is especially interesting in the context of graphic 
organizers. It is intriguing, for instance, to imagine what effect study of the GO prior to viewing the 
relevant text would have had on subjects’ task expectancy as well as processing precedence. 

Figure 6. Effect ofText-to-displav Semantic Coneruencv on Response Latencies for Answerins Questions 
According to Topic 
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Studies on how to effectively integrate cognitive tools, such as graphic organizers, with 
instructional design practices exemplify the kind of “prescriptive” research (Clark, 1989) that could 
contribute much to the field of educational technology. 

Graphic organizers are important adjuncts for enhancing what people recall from a related text. In 
many cases they yield insights on the relationship between facts in a story even better than the story itself 
To be effective, however, such learning tools should be designed so that they are semantically congruent 
with the texts they accompany. Doing so will maximize the mental processing of both. 
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